Local Models

Run PraisonAI against Ollama or any OpenAI-compatible local server — keyless

Point PraisonAI at Ollama, LM Studio, llama.cpp, or vLLM — no cloud key required.

from praisonaiagents import Agent

agent = Agent(
    name="Local Assistant",
    instructions="Answer briefly",
    llm="ollama/llama3.2",
)
agent.start("Why is the sky blue?")

See also: to skip the ollama/<model> prefix and let PraisonAI find and configure a running local server for you, set llm="local" — the explicit form. As of PR #4900, llm="local" correctly hands /v1 to LM Studio, vLLM, llama.cpp, and mlx-lm — earlier releases did not, and those engines returned 404 on chat. Keyless auto-detection remains the implicit form. See the Local Model Resolver.

Agent(llm="local", knowledge=[...]) on releases before PR #4870 silently sent documents and queries to OpenAI for embeddings. See Local Memory & Knowledge for the fix and how to verify.

Runtime accuracy since PR #4924. PraisonAI now uses the measurements the local layer already takes:

• Context window comes from the server (/api/show), not the litellm default — a 40960-token model is no longer budgeted at 128000. See Context Window Management.
• Embedding width reaches the vector store — nine more local embedders carry measured dimensions. See Local Memory & Knowledge.
• Optional[...] tool arguments work on Ollama / LM Studio / vLLM, and tools + output_json/output_pydantic is refused up front instead of silently fabricating an answer. See Local Tool Schemas.

Quick Start

1. Zero-config with Ollama

   Start Ollama, pull a model, and run — PraisonAI detects the endpoint when no cloud key is set.

   ollama serve
   ollama pull llama3.2
   praisonai run "hello"
   # > No cloud key found; using local model ollama/llama3.2.
   
2. Point at any OpenAI-compatible server

   Set OPENAI_BASE_URL to target LM Studio, llama.cpp, or vLLM.

   export OPENAI_BASE_URL=http://localhost:1234/v1
   praisonai run "hello"
   

   Paste the URL your server prints. http://localhost:1234/v1 (what LM Studio prints) and http://localhost:1234 are both handled — since PR #4870 the resolver strips a trailing /v1 before appending, so requests never become /v1/v1/models.

---

How It Works

PraisonAI probes for a reachable local server only when no cloud provider key is set, then adopts the first model it finds.

Server type	Probed path	Resolved model id
Ollama	GET /api/tags (at the root, not /v1/api/tags)	ollama/<first-model-name>
Generic OpenAI-compatible (llama.cpp, LM Studio, vLLM)	GET /v1/models (or GET /models when the base already ends in /v1)	openai/<first-id>

Since PR #4870, the /v1/models probe always runs and any 2xx counts as present. Even if GET / returns 404 (as llama-server, LM Studio and vLLM do), the /v1/models probe still runs and finds the server — earlier versions reported these servers as “did not answer (refused)” while they were actively serving.

Detection is timeout-bounded (~150 ms) and caches negatives for 30 s, so the first-run hot path stays fast when nothing is listening.

---

Helper model for internal calls

PraisonAI makes a small number of internal LLM calls for things like session-title generation, context compaction, memory quality scoring, and workflow routing. By default they use gpt-4o-mini. If your local server doesn’t serve a model called gpt-4o-mini (Ollama, LM Studio, and vLLM don’t) those calls will fail with a “model not found” error, even though the agent itself works fine.

Set PRAISONAI_AUXILIARY_MODEL to a small local model:

export OPENAI_BASE_URL=http://localhost:11434/v1
export PRAISONAI_AUXILIARY_MODEL=ollama/qwen3:0.6b
praisonai run "hello"

The complete four-line copy-paste example:

import os
from praisonaiagents import Agent

os.environ["OPENAI_BASE_URL"] = "http://localhost:11434/v1"
os.environ["OPENAI_MODEL_NAME"] = "ollama/llama3.3:70b"
os.environ["PRAISONAI_AUXILIARY_MODEL"] = "ollama/qwen3:0.6b"

agent = Agent(name="Local", instructions="Answer briefly")
agent.start("Explain black holes to a 5-year-old")

The auxiliary model is separate from your agent’s model. Point OPENAI_MODEL_NAME (or Agent(llm=…)) at your big local model for the agent’s own reasoning; point PRAISONAI_AUXILIARY_MODEL at a smaller one for the internal helper calls. The two do not need to be the same — that’s the whole point of the split (PR #4812).

Env var	Purpose	Example for fully-local Ollama
OPENAI_BASE_URL	Endpoint every OpenAI-compatible call reaches	http://localhost:11434/v1
OPENAI_MODEL_NAME	Primary agent model (falls through to auxiliary if PRAISONAI_AUXILIARY_MODEL is unset)	ollama/llama3.3:70b
PRAISONAI_AUXILIARY_MODEL	Internal helper model — session titles, compaction, memory quality, workflow routing	ollama/qwen3:0.6b
OPENAI_API_KEY	Local servers usually don’t need a key, but many reject an empty one. Any non-empty value works (e.g. local).	local

Which knob to set:

---

Configuration Options

Env var	Meaning	Default
OPENAI_BASE_URL	Any OpenAI-compatible base URL (with or without /v1)	—
OLLAMA_HOST	Ollama root (host:port allowed — a scheme is added if missing)	—
(none set)	Probe http://127.0.0.1:11434	—
PRAISONAI_HOME	Where session / credential data lives	~/.praisonai

Precedence (first match wins): --model <name> → any cloud provider key → reachable local endpoint (OPENAI_BASE_URL → OLLAMA_HOST → 127.0.0.1:11434) → gpt-4o-mini fallback.

---

Common Patterns

Ollama

ollama serve
ollama pull llama3.2
praisonai run "hello"

Pin a specific model explicitly with --model:

praisonai run "hello" --model ollama/llama3.2

---

Mixing local and hosted models

Setting a local base_url for local development is safe to combine with hosted model names — a base_url says where the server is, not what it serves.

from praisonaiagents import Agent

# Hosted model, behind a local base_url — safe; keeps DefaultAdapter.
gpt = Agent(
    name="hosted",
    instructions="Answer briefly",
    llm="gpt-4o",
    base_url="http://localhost:11434/v1",
    api_key="sk-...",   # real OpenAI key
)

# Same base_url, open-weights local model — gets OllamaAdapter with tool-call repair.
local = Agent(
    name="local",
    instructions="Answer briefly",
    llm="llama3.2",
    base_url="http://localhost:11434/v1",
)

Detection runs in a fixed order: an explicit ollama/ prefix wins, then closed-weights names keep their native adapter, then the base_url is checked, then the model name, then the default.

The ollama/ prefix is the user speaking explicitly — it wins over everything, even a closed-weights name:

from praisonaiagents import Agent

# Even a closed-weights-looking name is Ollama when the prefix says so.
agent = Agent(llm="ollama/gpt-oss")

Nested and vendor-qualified routes resolve to the family they name, so they keep their native adapter behind a local base_url too:

from praisonaiagents import Agent

agent = Agent(
    llm="bedrock/anthropic.claude-3-5-sonnet",
    base_url="http://localhost:11434/v1",   # safe: still routes to AnthropicAdapter
)

model form	with local base_url	adapter
ollama/… (explicit prefix)	any URL	OllamaAdapter
gpt-*, o1-*, o3-*, chatgpt-*	any URL	DefaultAdapter
claude*	any URL	AnthropicAdapter
gemini-*	any URL	GeminiAdapter
llama*, gemma*, mistral*, qwen*, deepseek*, phi*	local URL	OllamaAdapter (with tool-call repair)
llama*, gemma*, …	no URL	DefaultAdapter
nested routes (bedrock/anthropic.claude-*, openrouter/openai/gpt-*)	any URL	the family’s native adapter

---

Best Practices

Detection stays off the hot path

The probe runs only when no cloud key is set, times out at ~150 ms, and caches negatives for 30 s — a first run with nothing listening is not slowed down.

Cloud keys always win

Any cloud provider key (e.g. OPENAI_API_KEY, ANTHROPIC_API_KEY) skips the local probe entirely. OLLAMA_HOST is not a cloud key, so it participates in local detection instead.

Use --model to override detection

Pass --model ollama/llama3.2 (or any id) to bypass auto-detection and target an exact model.

Relocate state with PRAISONAI_HOME

Set PRAISONAI_HOME to move sessions, credentials, and cache in one place — handy for Nix, Docker, or Snap packaging. Defaults to ~/.praisonai.

---

Related

Provider Auto-Detection

The full precedence reference for model resolution.

First-run Onboarding

The complete credential resolution ladder.

Local Model Resolver

Set llm="local" to auto-discover and configure a running local server.