Local Model Resolver

Point an Agent at whatever local model server is running with llm=“local” — no URL, no model name, no key

Set llm="local" and the agent finds a running local model server, picks the best model on it, and configures itself — no URL, no model name, no key.

from praisonaiagents import Agent

agent = Agent(instructions="You are helpful", llm="local")
agent.start("Why is the sky blue?")

Discovery fires only when llm="local". Passing gpt-4o, ollama/llama3.2, or anthropic/… runs zero probes.

On releases before PR #4900, Agent(llm="local") handed every resolved engine the bare server root. That is correct for Ollama, but LM Studio, vLLM, llama-server and mlx-lm all serve under /v1 — so chat requests went to /chat/completions and those servers answered 404. Upgrade to a release that includes #4900 to use those engines via llm="local".

Quick Start

1. Discover whatever is running

   Start any local server (Ollama, llama.cpp, LM Studio, vLLM), then run four lines.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local")
   agent.start("Why is the sky blue?")
   

   No OPENAI_BASE_URL, no ollama/ prefix, no model name.

2. Pin an engine, let it pick the model

   Name the engine and the resolver chooses the best model on it.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local:ollama")
   agent.start("Explain gravity in one sentence")
   

   Swap ollama for llama_cpp, lm_studio, vllm, mlx_lm, or transformers_serve.

3. Pin engine and model

   Name both for reproducibility.

   from praisonaiagents import Agent
   
   agent = Agent(instructions="You are helpful", llm="local:ollama/qwen3:0.6b")
   agent.start("Summarize the water cycle")
   

---

How It Works

The agent calls the resolver, which probes loopback ports, identifies the engine, selects a model, and returns a target the normal LLM path uses unchanged.

The resolver never infers identity from a port — a server matches only when its probe rules hold. This prevents silently talking to the wrong server.

Base URL selection per engine

The base_url handed to litellm depends on the engine, because Ollama and the OpenAI-compatible engines expect different roots.

Engine	base_url handed to litellm	Reason
ollama	server root (e.g. http://127.0.0.1:11434)	litellm’s ollama provider builds its own /api paths from the root
lm_studio, vllm, llama_cpp, mlx_lm, transformers_serve	server root + /v1 (e.g. http://127.0.0.1:1234/v1)	These engines expose the OpenAI-compatible surface under /v1 — POST /chat/completions on the bare root returns 404

The resolver exposes both roots on LocalTarget as base_url (Ollama-shaped) and openai_base_url (OpenAI-shaped). The agent picks the right one per engine.

Precedence ladder

The resolver checks sources in a fixed order. The first three are authoritative: if one names a server that does not answer, resolution raises instead of quietly probing another port.

---

Which option should I pick?

Pick the shortest spec that gives you the certainty you need.

---

Supported local engines

Identity is decided by probe rules, never by the port — several engines share :8080 and :8000.

Engine (llm="local:<engine>")	Default port	Identity probe
ollama	11434	GET / returns Ollama is running
llama_cpp	8080	GET /health ok and GET /props has build_info
mlx_lm	8080	GET /health ok, GET /props absent, GET /v1/models has data
lm_studio	1234	GET /api/v0/models has data
vllm	8000	GET /version has version
transformers_serve	8000	GET /version absent, GET /health ok, GET /v1/models has data

llama_cpp and mlx_lm both default to :8080; vllm and transformers_serve both default to :8000. The “absent” rules (e.g. /props absent, /version absent) separate the co-tenants on each port.

---

Local embedder auto-selection

When Agent(llm="local", ...) needs embeddings (any of retrieval_config, embedder_config, or memory set) and no embedder is configured, the resolver picks one from the same local server.

Precedence:

1. PRAISONAI_LOCAL_EMBED_MODEL env var (validated against the server’s listing).
2. PREFERRED_EMBED_MODELS order: nomic-embed-text, mxbai-embed-large, bge-m3, snowflake-arctic-embed, all-minilm.
3. Any remaining embedding-capable model.

The server’s own capability list wins when available (Ollama exposes "embedding"); otherwise names containing embed or starting with bge- are treated as embedders.

The picker returns None when the server serves none — the agent then warns and never silently falls back to a remote provider.

Since PR #4870, a base URL with a path prefix is preserved end-to-end: OLLAMA_HOST=http://gateway:8000/ollama keeps the /ollama prefix on every request.

Since PR #4924, the measurements the resolver already takes now drive behaviour: the picked embedder carries its measured embedding_dims to the vector store, and the model’s context window comes from the server rather than the litellm default. Tool schemas are made server-safe per engine — see Local Tool Schemas for the Optional[...] collapse and the tools + structured-output refusal.

Advanced: build the block yourself

The same helpers are exported from praisonaiagents.local:

from praisonaiagents.local import (
    select_embedding_model, local_embedder_config, discover,
)

discovery = discover()   # finds a running local server
model = select_embedding_model(discovery.models, discovery.model_meta)
embedder_block = local_embedder_config(discovery.engine, discovery.base_url, model)

A resolved LocalTarget exposes both roots: base_url (Ollama-shaped, bare root) and openai_base_url (OpenAI-shaped, ends in /v1). Read openai_base_url when wiring an OpenAI-compatible engine (LM Studio, vLLM, llama.cpp, mlx-lm) yourself.

---

Environment Variables

Every source is optional. The three authoritative sources raise when they name an unreachable server.

Variable	Purpose	Default
PRAISONAI_LOCAL_BASE_URL	Authoritative base URL — if set and unreachable, resolution raises	(none)
PRAISONAI_LOCAL_ENGINE	Pin the expected engine (ollama, llama_cpp, lm_studio, vllm, …)	(none)
PRAISONAI_LOCAL_MODEL	Pin the model id	(none)
PRAISONAI_LOCAL_EMBED_MODEL	Pin the embedding model the resolver picks for knowledge / memory. Accepts a bare name (all-minilm) or a tagged id (all-minilm:latest). Falls back to PREFERRED_EMBED_MODELS order when unset.	(none)
PRAISONAI_LOCAL_TTL	Positive-result cache seconds	30
PRAISONAI_LOCAL_NEG_TTL	Negative-result cache seconds	5
PRAISONAI_LOCAL_TIMEOUT	Total resolve budget in seconds (clamped 0.05–30; per-probe is 0.4)	1.5
OLLAMA_HOST	Authoritative Ollama host (see the port-80 gotcha below)	(none)
OPENAI_BASE_URL / OPENAI_API_BASE	Honoured only when the host is local (127.0.0.1, localhost, private)	(none)

Engines without a built-in ProbeSpec (llamafile, localai, ramalama) are reachable only via PRAISONAI_LOCAL_ENGINE — since PR #4870 they resolve through the GET /v1/models fallback probe.

---

How model selection works

When you do not name a model, the resolver ranks the server’s models and picks the best one to chat with.

• Tools-capable models rank first — an agent that can call tools is preferred.
• Embedding-only models rank last — an embedder cannot hold a conversation, so it is never picked by accident.
• Ties break by recency — among equally-capable models, the newest wins.

Worked example. On a machine serving qwen3:0.6b, all-minilm, nomic-embed-text, and mxbai-embed-large, the resolver picks qwen3:0.6b — even when the embedders are newer — because the three embed-only models are ranked last.

An explicitly named model must actually be served. llm="local:ollama/does-not-exist" raises ModelNotAvailableError and lists what is available, rather than sending a phantom model to the server.

---

Errors and gotchas

The resolver fails loudly and specifically instead of guessing.

NoLocalEngineError — nothing answered

No server responded on any probed port. Start one (ollama serve, then ollama pull qwen3:0.6b) or set PRAISONAI_LOCAL_BASE_URL to its address.

EngineUnreachableError — a named server is silent

An authoritative source (the spec, PRAISONAI_LOCAL_BASE_URL, or OLLAMA_HOST) named a server that did not answer. Because it was named explicitly, the resolver does not fall through to another port — it raises so you fix the real target.

HostHeaderRejectedError — Ollama refused the Host header

The server returned a bodyless HTTP 403. Ollama rejects any request whose Host header is not localhost or an IP address. Set OLLAMA_HOST on the server to allow the origin.

ModelNotAvailableError — that model is not served

A specific model was requested but the engine does not have it. Pull it or name one from the list the error prints.

InvalidLocalSpecError — the spec could not be parsed

The llm="local:..." string was malformed. Expected forms: "local", "local:<engine>", "local:<engine>/<model>", a base URL, or "<url>#<model>".

OLLAMA_HOST port-80 trap

Setting OLLAMA_HOST to a scheme with no port means port 80, not 11434. The resolver names this exact trap in the error:

Local runtime at http://127.0.0.1:80 did not answer (refused). It was named
explicitly by OLLAMA_HOST='http://127.0.0.1', so no other port was probed.
Note that OLLAMA_HOST with an explicit http:// scheme and no port means port
80, not 11434.

Use OLLAMA_HOST=127.0.0.1:11434 (bare host defaults to 11434) instead.

---

Performance notes

Resolution is opt-in and cheap.

• Discovery fires only for llm="local" — cloud models trigger zero probes.
• Importing praisonaiagents does not import the resolver, so import time is unchanged.
• A full resolve issues a handful of loopback HTTP requests and caches the result for 30 s (PRAISONAI_LOCAL_TTL).
• The whole scan is budgeted (PRAISONAI_LOCAL_TIMEOUT, default 1.5 s; per-probe 0.4 s), so nothing stalls when no server is listening.

---

Explicit vs implicit local

llm="local" is the explicit request — you ask for a local model by name. The keyless local-first fallback is the implicit form — it only kicks in when no cloud key is set and you named no model at all.

	Explicit llm="local"	Implicit keyless fallback
Trigger	You pass llm="local"	No cloud key and no --model
Silent fallthrough	Never — authoritative sources raise	Falls back to gpt-4o-mini
Engine choice	Full probe table (Ollama, llama.cpp, LM Studio, vLLM, mlx_lm, transformers serve)	Ollama /api/tags, then /v1/models

---

Related

Models

Provider auto-detection and the full model-selection precedence.

Keyless Local-First Run

The implicit local fallback when no cloud key is set.

Local Models

Point PraisonAI at Ollama or any OpenAI-compatible server.

Ollama

Use Ollama models with PraisonAI.