> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Local Models > Run PraisonAI against Ollama or any OpenAI-compatible local server — keyless Point PraisonAI at Ollama, LM Studio, llama.cpp, or vLLM — no cloud key required. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( name="Local Assistant", instructions="Answer briefly", llm="ollama/llama3.2", ) agent.start("Why is the sky blue?") ``` **See also:** to skip the `ollama/` prefix and let PraisonAI find and configure a running local server for you, set `llm="local"` — the explicit form. As of PR [#4900](https://github.com/MervinPraison/PraisonAI/pull/4900), `llm="local"` correctly hands `/v1` to LM Studio, vLLM, llama.cpp, and mlx-lm — earlier releases did not, and those engines returned 404 on chat. Keyless auto-detection remains the implicit form. See the [Local Model Resolver](/docs/features/local-model-resolver). `Agent(llm="local", knowledge=[...])` on releases before PR [#4870](https://github.com/MervinPraison/PraisonAI/pull/4870) silently sent documents and queries to OpenAI for embeddings. See [Local Memory & Knowledge](/docs/features/local-memory-and-knowledge) for the fix and how to verify. **Runtime accuracy since PR [#4924](https://github.com/MervinPraison/PraisonAI/pull/4924).** PraisonAI now uses the measurements the local layer already takes: * **Context window** comes from the server (`/api/show`), not the litellm default — a 40960-token model is no longer budgeted at 128000. See [Context Window Management](/docs/features/context-window-management#local-models-window-comes-from-the-server). * **Embedding width** reaches the vector store — nine more local embedders carry measured dimensions. See [Local Memory & Knowledge](/docs/features/local-memory-and-knowledge#vector-store-is-now-sized-correctly). * **`Optional[...]` tool arguments** work on Ollama / LM Studio / vLLM, and `tools` + `output_json`/`output_pydantic` is refused up front instead of silently fabricating an answer. See [Local Tool Schemas](/docs/features/local-tool-schemas). ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Local Models" User[📋 User] --> Agent[🤖 Agent] Agent --> Detect[🔍 Detect local endpoint] Detect --> Server[✅ Local server] end classDef input fill:#8B0000,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef output fill:#10B981,stroke:#7C90A0,color:#fff class User,Agent input class Detect process class Server output ``` ## Quick Start Start Ollama, pull a model, and run — PraisonAI detects the endpoint when no cloud key is set. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} ollama serve ollama pull llama3.2 praisonai run "hello" # > No cloud key found; using local model ollama/llama3.2. ``` Set `OPENAI_BASE_URL` to target LM Studio, llama.cpp, or vLLM. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export OPENAI_BASE_URL=http://localhost:1234/v1 praisonai run "hello" ``` Paste the URL your server prints. `http://localhost:1234/v1` (what LM Studio prints) and `http://localhost:1234` are both handled — since PR [#4870](https://github.com/MervinPraison/PraisonAI/pull/4870) the resolver strips a trailing `/v1` before appending, so requests never become `/v1/v1/models`. *** ## How It Works PraisonAI probes for a reachable local server only when no cloud provider key is set, then adopts the first model it finds. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant CLI participant Detect as detect_local_model participant Server as Local Server CLI->>Detect: no cloud key — probe? Detect->>Server: GET /api/tags Server-->>Detect: models[0].name → ollama/ Detect->>Server: GET /v1/models (or /models if base ends in /v1) Server-->>Detect: any 2xx → present; data[0].id → openai/ Detect-->>CLI: LocalModel(model, base_url) classDef cli fill:#8B0000,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef output fill:#10B981,stroke:#7C90A0,color:#fff class CLI cli class Detect process class Server output ``` | Server type | Probed path | Resolved model id | | ------------------------------------------------------ | ----------------------------------------------------------------------- | --------------------------- | | Ollama | `GET /api/tags` (at the root, not `/v1/api/tags`) | `ollama/` | | Generic OpenAI-compatible (llama.cpp, LM Studio, vLLM) | `GET /v1/models` (or `GET /models` when the base already ends in `/v1`) | `openai/` | Since PR [#4870](https://github.com/MervinPraison/PraisonAI/pull/4870), the `/v1/models` probe always runs and **any 2xx counts as present**. Even if `GET /` returns 404 (as `llama-server`, LM Studio and vLLM do), the `/v1/models` probe still runs and finds the server — earlier versions reported these servers as "did not answer (refused)" while they were actively serving. Detection is timeout-bounded (\~150 ms) and caches negatives for 30 s, so the first-run hot path stays fast when nothing is listening. *** ## Helper model for internal calls PraisonAI makes a small number of *internal* LLM calls for things like session-title generation, context compaction, memory quality scoring, and workflow routing. By default they use `gpt-4o-mini`. If your local server doesn't serve a model called `gpt-4o-mini` (Ollama, LM Studio, and vLLM don't) those calls will fail with a "model not found" error, even though the agent itself works fine. Set `PRAISONAI_AUXILIARY_MODEL` to a small local model: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export OPENAI_BASE_URL=http://localhost:11434/v1 export PRAISONAI_AUXILIARY_MODEL=ollama/qwen3:0.6b praisonai run "hello" ``` The complete four-line copy-paste example: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import Agent os.environ["OPENAI_BASE_URL"] = "http://localhost:11434/v1" os.environ["OPENAI_MODEL_NAME"] = "ollama/llama3.3:70b" os.environ["PRAISONAI_AUXILIARY_MODEL"] = "ollama/qwen3:0.6b" agent = Agent(name="Local", instructions="Answer briefly") agent.start("Explain black holes to a 5-year-old") ``` ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant Helper as Internal helper participant Local as Local server (Ollama :11434) User->>Agent: agent.start("Explain black holes...") Agent->>Local: OPENAI_MODEL_NAME=ollama/llama3.3:70b Local-->>Agent: reasoning + response Agent->>Helper: summarise for session title / compact context Helper->>Local: PRAISONAI_AUXILIARY_MODEL=ollama/qwen3:0.6b Local-->>Helper: short summary Helper-->>Agent: done Agent-->>User: response ``` The auxiliary model is *separate* from your agent's model. Point `OPENAI_MODEL_NAME` (or `Agent(llm=…)`) at your big local model for the agent's own reasoning; point `PRAISONAI_AUXILIARY_MODEL` at a smaller one for the internal helper calls. The two do not need to be the same — that's the whole point of the split ([PR #4812](https://github.com/MervinPraison/PraisonAI/pull/4812)). | Env var | Purpose | Example for fully-local Ollama | | --------------------------- | --------------------------------------------------------------------------------------------------------------- | ------------------------------ | | `OPENAI_BASE_URL` | Endpoint every OpenAI-compatible call reaches | `http://localhost:11434/v1` | | `OPENAI_MODEL_NAME` | Primary agent model (falls through to auxiliary if `PRAISONAI_AUXILIARY_MODEL` is unset) | `ollama/llama3.3:70b` | | `PRAISONAI_AUXILIARY_MODEL` | Internal helper model — session titles, compaction, memory quality, workflow routing | `ollama/qwen3:0.6b` | | `OPENAI_API_KEY` | Local servers usually don't need a key, but many reject an empty one. Any non-empty value works (e.g. `local`). | `local` | Which knob to set: ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB Start[Need to set the auxiliary model] --> Q1{Using .praisonai/config.toml?} Q1 -->|Yes| A[Set defaults.small_model] Q1 -->|No| Q2{Same model for agent AND helpers?} Q2 -->|Yes| B[Set OPENAI_MODEL_NAME only] Q2 -->|No| C[Set PRAISONAI_AUXILIARY_MODEL for helpers,
OPENAI_MODEL_NAME for agent] classDef question fill:#F59E0B,stroke:#7C90A0,color:#fff classDef answer fill:#10B981,stroke:#7C90A0,color:#fff classDef start fill:#6366F1,stroke:#7C90A0,color:#fff class Start start class Q1,Q2 question class A,B,C answer ``` *** ## Configuration Options | Env var | Meaning | Default | | ----------------- | ---------------------------------------------------------------- | -------------- | | `OPENAI_BASE_URL` | Any OpenAI-compatible base URL (with or without `/v1`) | — | | `OLLAMA_HOST` | Ollama root (`host:port` allowed — a scheme is added if missing) | — | | (none set) | Probe `http://127.0.0.1:11434` | — | | `PRAISONAI_HOME` | Where session / credential data lives | `~/.praisonai` | Precedence (first match wins): `--model ` → any cloud provider key → reachable local endpoint (`OPENAI_BASE_URL` → `OLLAMA_HOST` → `127.0.0.1:11434`) → `gpt-4o-mini` fallback. *** ## Common Patterns ```bash Ollama theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} ollama serve ollama pull llama3.2 praisonai run "hello" ``` ```bash LM Studio theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export OPENAI_BASE_URL=http://localhost:1234/v1 praisonai run "hello" ``` ```bash llama.cpp theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export OPENAI_BASE_URL=http://localhost:8080/v1 praisonai run "hello" ``` ```bash vLLM theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} export OPENAI_BASE_URL=http://localhost:8000/v1 praisonai run "hello" ``` Pin a specific model explicitly with `--model`: ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai run "hello" --model ollama/llama3.2 ``` *** ## Mixing local and hosted models Setting a local `base_url` for local development is safe to combine with hosted model names — a `base_url` says *where* the server is, not *what* it serves. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent # Hosted model, behind a local base_url — safe; keeps DefaultAdapter. gpt = Agent( name="hosted", instructions="Answer briefly", llm="gpt-4o", base_url="http://localhost:11434/v1", api_key="sk-...", # real OpenAI key ) # Same base_url, open-weights local model — gets OllamaAdapter with tool-call repair. local = Agent( name="local", instructions="Answer briefly", llm="llama3.2", base_url="http://localhost:11434/v1", ) ``` Detection runs in a fixed order: an explicit `ollama/` prefix wins, then closed-weights names keep their native adapter, then the `base_url` is checked, then the model name, then the default. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Agent participant Detect as _detect_provider participant Guard as is_hosted_only_model participant Adapter Agent->>Detect: model + base_url Detect->>Detect: ollama/ prefix? → OllamaAdapter Detect->>Guard: closed-weights name? Guard-->>Detect: yes → keep native adapter Detect->>Detect: local base_url? → OllamaAdapter Detect->>Detect: claude / gemini in name? → native Detect->>Adapter: else → DefaultAdapter Note over Agent,Adapter: gpt-4o + local base_url → DefaultAdapter
llama3.2 + local base_url → OllamaAdapter ``` The `ollama/` prefix is the user speaking explicitly — it wins over everything, even a closed-weights name: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent # Even a closed-weights-looking name is Ollama when the prefix says so. agent = Agent(llm="ollama/gpt-oss") ``` Nested and vendor-qualified routes resolve to the family they name, so they keep their native adapter behind a local `base_url` too: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent agent = Agent( llm="bedrock/anthropic.claude-3-5-sonnet", base_url="http://localhost:11434/v1", # safe: still routes to AnthropicAdapter ) ``` | model form | with local `base_url` | adapter | | ----------------------------------------------------------------------- | --------------------- | --------------------------------------- | | `ollama/…` (explicit prefix) | any URL | `OllamaAdapter` | | `gpt-*`, `o1-*`, `o3-*`, `chatgpt-*` | any URL | `DefaultAdapter` | | `claude*` | any URL | `AnthropicAdapter` | | `gemini-*` | any URL | `GeminiAdapter` | | `llama*`, `gemma*`, `mistral*`, `qwen*`, `deepseek*`, `phi*` | local URL | `OllamaAdapter` (with tool-call repair) | | `llama*`, `gemma*`, … | no URL | `DefaultAdapter` | | nested routes (`bedrock/anthropic.claude-*`, `openrouter/openai/gpt-*`) | any URL | the family's native adapter | *** ## Best Practices The probe runs only when no cloud key is set, times out at \~150 ms, and caches negatives for 30 s — a first run with nothing listening is not slowed down. Any cloud provider key (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`) skips the local probe entirely. `OLLAMA_HOST` is not a cloud key, so it participates in local detection instead. Pass `--model ollama/llama3.2` (or any id) to bypass auto-detection and target an exact model. Set `PRAISONAI_HOME` to move sessions, credentials, and cache in one place — handy for Nix, Docker, or Snap packaging. Defaults to `~/.praisonai`. *** ## Related The full precedence reference for model resolution. The complete credential resolution ladder. Set `llm="local"` to auto-discover and configure a running local server.