Model Fallback

Keep your agent answering when a model is overloaded by automatically falling back to alternates

Model Fallback keeps your agent answering by automatically retrying on alternate models when the primary model is overloaded or unavailable.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    name="assistant",
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["anthropic/claude-3-5-sonnet", "gpt-4o-mini"],
    ),
)
agent.start("Answer even when the primary model is overloaded")

The user sends a prompt; the agent retries on fallback models when the primary returns errors.

Quick Start

1. One-line resilience

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   
   agent = Agent(
       instructions="You are a helpful assistant",
       model=LLMConfig(
           model="gpt-4o",
           fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
       ),
   )
   agent.start("Summarise today's news")
   
2. Cross-provider chain

   Use LiteLLM-style prefixes when mixing providers:

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   
   agent = Agent(
       model=LLMConfig(
           model="openai/gpt-4o",
           fallback_models=["anthropic/claude-3-5-sonnet", "openai/gpt-4o-mini"],
       ),
   )
   
3. Notice the switch

   Subscribe to HookEvent.MODEL_FALLBACK to react the moment the primary model is swapped out:

   from praisonaiagents import Agent
   from praisonaiagents.config import LLMConfig
   from praisonaiagents.hooks import HookRegistry, HookEvent, HookResult
   
   registry = HookRegistry()
   
   @registry.on(HookEvent.MODEL_FALLBACK)
   def notice(evt):
       print(f"[fallback] {evt.from_model} → {evt.to_model} ({evt.reason_category})")
       return HookResult.allow()
   
   agent = Agent(
       instructions="You are a helpful assistant",
       model=LLMConfig(
           model="gpt-4o",
           fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
       ),
       hooks=registry,
   )
   agent.start("Answer even during a provider outage")
   

How It Works

On transient errors (503, timeout, model overloaded), the agent retries the same turn against the next model in fallback_models. Successful calls stay on the primary model.

Failover fires on retryable errors classified by the LLM error classifier and covers every turn shape — non-streaming, streaming, tool-iteration turns, reflection turns, and their async equivalents. A 503 on a streaming chunk pushes the same turn to the next model in fallback_models; the user sees continuous output, not a failure.

Observing the Switch

Subscribe to HookEvent.MODEL_FALLBACK (or the MODEL_FALLBACK stream event) to react the moment the primary model is swapped out — for alerts, metrics, a UI notice, or a per-user notification.

MODEL_FALLBACK is dispatched through the agent-scoped hook registry passed via Agent(hooks=...). Register on the agent’s registry (as shown below) so the hook fires on both sync and async runs. Hooks registered only on the global default registry may be skipped on the async path.

ModelFallbackInput fields

Field	Type	Default	Description
from_model	str	""	Model id the turn was using when the failure hit (e.g. "gpt-4o").
to_model	str	""	Next entry in the fallback_models chain that the turn continues on.
reason_category	str	""	Failure classification from the LLM error classifier (e.g. "rate_limit", "provider_down", "timeout"). Provider internals stay redacted.
fallback_index	int	0	0-based index into fallback_models of the entry now in use.
session_id, cwd, event_name, timestamp, agent_name	—	—	Standard HookInput fields.

Notification only — the turn already continued on to_model. Provider internals are redacted; only the failure class reaches your hook. Zero overhead when unsubscribed. Errors inside the hook never break the fallback path.

Hook subscription

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig
from praisonaiagents.hooks import HookRegistry, HookEvent, HookResult

registry = HookRegistry()

@registry.on(HookEvent.MODEL_FALLBACK)
def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    print(f"           reason={evt.reason_category} index={evt.fallback_index}")
    return HookResult.allow()

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
    hooks=registry,
)
agent.start("Answer even during a provider outage")

Stream event

When a stream callback is active, the same swap emits a StreamEventType.MODEL_FALLBACK event carrying the four fields in metadata, plus agent_id, session_id, and run_id.

The stream event honours the run’s emit_events flag: when events are suppressed, StreamEventType.MODEL_FALLBACK is skipped, but HookEvent.MODEL_FALLBACK still fires. Use the hook when you need a guaranteed signal regardless of stream configuration.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig
from praisonaiagents.streaming import StreamEvent, StreamEventType

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
    stream=True,
)

def on_event(event: StreamEvent):
    if event.type == StreamEventType.MODEL_FALLBACK:
        m = event.metadata
        print(f"[stream] switched {m['from_model']} → {m['to_model']} ({m['reason_category']})")

agent.stream_emitter.add_callback(on_event)
agent.start("Answer even during a provider outage")

Sync and async paths

Both call sites emit — the sync _chat_completion path and the async _handle_async_llm_error path — so the hook fires whether you run agent.start(...) or await agent.astart(...).

Sync path

@registry.on(HookEvent.MODEL_FALLBACK)
def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    return HookResult.allow()

agent.start("Answer even during a provider outage")

Async path

@registry.on(HookEvent.MODEL_FALLBACK)
async def on_fallback(evt):
    print(f"[fallback] {evt.from_model} → {evt.to_model}")
    return HookResult.allow()

await agent.astart("Answer even during a provider outage")

Async hooks are awaited (not fire-and-forget). The fallback retry does not proceed until the hook returns, so long-running notifiers should keep their work non-blocking or offload it.

Shared-Agent async safety

On a shared Agent, a fallback on one turn does not affect the model, cost, or observability attribution of concurrent turns.

Many users share a single Agent across asyncio.gather (see Thread-Safe Agent State → Chat History). Before this fix, a mid-fallback shared-state mutation could silently flip a concurrent turn to the fallback model — wrong cost, wrong logs, wrong hook payloads. Both paths now keep each turn on its own model.

import asyncio
from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are helpful.",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet"],
    ),
)

async def main():
    # Same Agent, two concurrent turns.
    # If turn A hits a 503 and falls back to claude-3-5-sonnet,
    # turn B still dispatches on gpt-4o with correct cost/observability.
    await asyncio.gather(
        agent.astart("Summarise today's news"),
        agent.astart("Draft a follow-up email"),
    )

asyncio.run(main())

Async path stays clean. The fallback retry runs against a throwaway per-call dispatcher bound to the fallback model; the shared model state is never mutated, so a concurrent turn keeps dispatching on its original model even while this turn is suspended on an await. No lock is taken on the async path for ordinary turns — non-fallback turns stay fully concurrent.

Sync path stays clean. The sync fallback still recurses through the full chat wrapper (which reads the current model for BEFORE_LLM hook attribution and budget), so it temporarily swaps the model. That mutate/restore window is now serialised by a lazily-created per-instance lock; only fallback turns contend on it — non-fallback turns never acquire it.

Custom-LLM caveat. For custom-LLM agents, the async fallback path continues to use the shared dispatcher (the throwaway builder returns None), matching prior behaviour, because the custom client owns its own model.

No new configuration or lock code is required on your side — share one Agent under asyncio.gather and both fallback and chat-history stay correct. Reference: PR #4593, issue #4592.

Configuration Options

Model Fallback is configured through LLMConfig — set model for the primary and fallback_models for the ordered backup chain.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    instructions="You are a helpful assistant",
    model=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
)
agent.start("Hello!")

Option	Type	Default	Description
model	str	—	Primary model the turn starts on.
fallback_models	Optional[List[str]]	None	Ordered backup chain; the runtime advances to the next entry on a retryable failure.

The decision diagram below shows which chain shape fits which scenario.

Sharing an Agent across concurrent turns? Both sync and async paths keep concurrent turns’ model attribution correct — see Shared-Agent async safety above.

Streaming, tool-iteration and reflection turns route through the same failover engine as plain single-shot calls (unified in PraisonAI PR #2665). You don’t need a separate fallback_models= setting for streaming paths.

LLMConfig API Reference

Full LLMConfig surface is covered by the auto-generated SDK reference — see Hook Events for the paired MODEL_FALLBACK hook.

Best Practices

Put a cheap same-provider fallback last

Useful for rate limits, not full provider outages — a cheap model on the same API may still fail if the provider is down.

Order by latency and cost

Fallback runs the same prompt; a much weaker model may return a worse answer, not a missing one.

Limit chain length to 2–3

Longer chains delay user-visible errors without improving success rates much.

Use provider prefixes when mixing

LiteLLM-style names (anthropic/..., openai/...) route credentials correctly across providers.

Related

LLM Configuration

Endpoints, API keys, and auth headers.

Models

Choosing models for agents.

Model Router

Dynamic model selection policies.

Rate Limiter

Throttle requests before they fail.

Thread-Safe Agent State

Share one Agent across concurrent turns safely.

Concurrency

Run agents in parallel with asyncio.gather.