Offline Testing

Unit-test agents without hitting a provider

Run agent tests offline — no API keys, no network, no bill — with ScriptedModel in place of a real provider and allow_model_requests(False) as a hard gate.

from praisonaiagents import Agent, ScriptedModel

model = ScriptedModel(["Paris."])
agent = Agent(instructions="You are a geography bot.", llm=model)

assert agent.start("What is the capital of France?") == "Paris."
assert model.requests[0].last_user_message == "What is the capital of France?"

ScriptedModel subclasses the real LLM, so everything around the call — system prompt assembly, tool schemas, the tool loop, streaming, async — runs production code; only the provider-facing methods are replaced.

Quick Start

1. Install and gate the suite

   Add the gate to conftest.py so nothing in the suite can reach a provider.

   pip install praisonaiagents
   

   # conftest.py
   from praisonaiagents import allow_model_requests
   
   allow_model_requests(False)   # nothing in this suite may reach a provider
   
2. Script one text reply

   from praisonaiagents import Agent, ScriptedModel
   
   model = ScriptedModel(["Paris."])
   agent = Agent(instructions="Geography bot.", llm=model)
   
   assert agent.start("Capital of France?") == "Paris."
   
3. Script a tool call + follow-up

   from praisonaiagents import Agent, ScriptedModel
   
   def refund(order_id: str) -> str:
       return f"refunded {order_id}"
   
   model = ScriptedModel([
       ScriptedModel.tool_call("refund", {"order_id": "A1"}),
       "Refunded order A1.",
   ])
   agent = Agent(instructions="Support bot.", llm=model, tools=[refund])
   
   assert agent.start("refund order A1") == "Refunded order A1."
   
4. Assert on what the agent sent

   assert model.request_count == 2                 # tool turn, then the follow-up
   assert model.requests[0].tool_names == ("refund",)   # tools offered on turn 1
   assert model.requests[0].last_user_message == "refund order A1"
   

Canonical example

# conftest.py
from praisonaiagents import allow_model_requests
allow_model_requests(False)          # nothing in this suite may reach a provider

from praisonaiagents import Agent, ScriptedModel

def refund(order_id: str) -> str:
    return f"refunded {order_id}"

model = ScriptedModel([
    ScriptedModel.tool_call("refund", {"order_id": "A1"}),
    "Refunded order A1.",
])
agent = Agent(instructions="Support bot.", llm=model, tools=[refund])

assert agent.start("refund order A1") == "Refunded order A1."
assert model.request_count == 2
assert model.requests[0].tool_names == ("refund",)

How it works

ScriptedModel is a real LLM subclass that replaces only the methods which talk to a provider — nothing else.

• System prompts are assembled normally, tools are serialised to real schemas, scripted tool calls are dispatched through the agent’s own executor, and results are fed back as real tool messages.
• Replies are built as genuine litellm.ModelResponse objects, parsed by the same code path a live response would take — so the double cannot drift from reality.
• When the agent asks for one more reply than the script holds, the double raises ScriptExhausted (usually the script is one reply short — after a tool call the agent comes back for a follow-up answer).

A script entry may be a str (a final answer), a ScriptedModel.tool_call(...), a list of tool calls, a ScriptedReply, or a callable receiving the RecordedRequest:

from praisonaiagents import ScriptedModel

# A reply that depends on what the agent actually sent:
model = ScriptedModel([lambda req: f"You said: {req.last_user_message}"])

The request gate

allow_model_requests(False) blocks all six litellm request sites and both OpenAI client properties process-wide. Any attempt to reach a provider raises ModelRequestBlocked, whose message names the offending call site in your code.

from praisonaiagents import Agent, allow_model_requests, ModelRequestBlocked

allow_model_requests(False)

agent = Agent(instructions="…", llm="gpt-4o")   # a real model, no script
try:
    agent.start("hello")
except ModelRequestBlocked as e:
    print(e.provider)    # e.g. "litellm.completion"
    print(e.call_site)   # "<your_test>.py:42 in test_thing" — the line to fix

Call allow_model_requests(True) to restore normal behaviour, or scope a single block with no_model_requests():

from praisonaiagents import Agent, no_model_requests

with no_model_requests():
    Agent(instructions="…", llm="gpt-4o").start("hello")   # raises ModelRequestBlocked

ModelRequestBlocked and ScriptExhausted derive from BaseException, not Exception — on purpose. The agent’s tool loop catches Exception broadly and turns failures into a None answer. An ordinary exception raised from inside the double would reach your test as a mysterious None, hiding the very thing the test needs to be told (“your script is one reply short”, “a real request leaked”). Deriving from BaseException lets these escape the loop — the same reasoning pytest uses for its own outcome exceptions. Do not wrap your agent calls in a bare except Exception that would swallow them.

When to use offline vs. real-API testing

Offline tests assert what the agent does — which tool it calls, in what order, with what arguments. For prompt quality and model behaviour, use Real API Testing.

ScriptedModel configuration

Option	Type	Default	Description
script	Iterable[str | ScriptedReply | ScriptedToolCall | dict | callable]	()	Ordered replies. Strings are text answers; use ScriptedModel.tool_call(...) for a tool call; a callable receives the RecordedRequest.
model	str (keyword-only)	"scripted/model"	Fake model id reported by the agent. Set a real id (e.g. "gpt-4o") to exercise model-specific behaviour while still answering from the script.
**llm_kwargs	—	—	Forwarded to the underlying LLM.

Inspection surface after a run:

Attribute / method	Returns	Use
model.request_count	int	How many turns the agent sent.
model.requests	tuple[RecordedRequest, …]	Every request, in order.
requests[i].tool_names	tuple[str, …]	Tool schemas offered on that turn.
requests[i].last_user_message	str | None	Most recent user message text.
requests[i].system_prompt	str | None	The assembled system prompt.
requests[i].params	dict	Every completion param sent (tool_choice, response_format, …).
requests[i].tool_results	tuple[dict, …]	Tool-result messages in that request’s history.

ScriptedReply, ScriptedToolCall, and RecordedRequest are importable from praisonaiagents.model_harness when you need to build or type them explicitly.

Best Practices

Gate the whole session in a fixture

Put allow_model_requests(False) in a session-scoped fixture (or conftest.py) so a forgotten mock cannot leak a real call and quietly bill you.

Script the arguments the agent will really produce

Assert requests[i] reflects the call you expect — a scripted tool_call("refund", {"order_id": "A1"}) should mirror what your instructions steer the model toward.

Prefer real ModelResponse shapes

Script with strings and ScriptedModel.tool_call(...) rather than inventing raw provider dicts — the helpers build genuine litellm response objects that the production parser understands.

Never mix scripted and real in one session

Keep offline tests and RUN_REAL_KEY_TESTS integration tests in separate sessions. allow_model_requests(False) blocks the whole process, so a real-API test in the same run would fail with ModelRequestBlocked.

Related

Real API Testing

Gated integration tests against live providers.

Guardrails

Test guardrail wiring offline with scripted replies.

Tools

How the tool loop the double exercises works.