> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Failover > Automatic fallback between LLM providers for reliability and cost optimization Failover automatically switches between LLM providers when one fails, keeping agents running during API outages or rate limits. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import Agent, AuthProfile, FailoverConfig, FailoverManager manager = FailoverManager(FailoverConfig(max_retries=3)) manager.add_profile(AuthProfile( name="openai", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=1, )) manager.add_profile(AuthProfile( name="anthropic", provider="anthropic", api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2, )) agent = Agent(name="assistant", llm={"model": "gpt-4o-mini", "failover_manager": manager}) agent.start("Hello!") ``` ```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import { FailoverManager } from 'praisonai'; const manager = new FailoverManager({ providers: ['openai', 'anthropic'], strategy: 'priority', maxRetries: 3, }); let provider = manager.getNextProvider(); // 'openai' under the priority strategy // Rotate on a retryable failure (e.g. a 401 that a fresh key may fix): manager.markFailed(provider!); provider = manager.getNextProvider(); // now falls through to 'anthropic' manager.markHealthy('openai', 120); // back online, 120ms latency console.log(manager.getStatuses()); ``` The user sends a message; if the primary LLM profile fails, the agent tries the next profile in the failover chain. The LLM tool-loop failover rotates profiles on a retryable auth failure (a `401` that a fresh key may fix) and fires an event hook carrying `{ attempt, maxAttempts, profile, retryInSeconds }` so you can log or alert on each rotation. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Failover Chain" A[πŸ€– Agent] --> P1[🟒 Primary] P1 -->|Fail| P2[🟑 Secondary] P2 -->|Fail| P3[πŸ”΅ Tertiary] P3 --> R[βœ… Response] end classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff classDef primary fill:#10B981,stroke:#7C90A0,color:#fff classDef secondary fill:#F59E0B,stroke:#7C90A0,color:#fff classDef tertiary fill:#6366F1,stroke:#7C90A0,color:#fff classDef result fill:#10B981,stroke:#7C90A0,color:#fff class A agent class P1 primary class P2 secondary class P3 tertiary class R result classDef tool fill:#189AB4,color:#fff classDef agent fill:#8B0000,color:#fff ``` Failover integrates with [LLM Error Classification](/docs/features/llm-error-classification) via `FailoverDecision`, which coordinates profile rotation with typed error handling. ## Quick Start ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import Agent, AuthProfile, FailoverConfig, FailoverManager manager = FailoverManager(FailoverConfig(max_retries=3, exponential_backoff=True)) manager.add_profile(AuthProfile( name="openai", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=1, )) manager.add_profile(AuthProfile( name="anthropic", provider="anthropic", api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2, )) agent = Agent( name="assistant", llm={"model": "gpt-4o-mini", "failover_manager": manager}, ) agent.start("Hello!") ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import AuthProfile, FailoverManager manager = FailoverManager() manager.add_profile(AuthProfile( name="groq", provider="groq", api_key=os.getenv("GROQ_API_KEY"), priority=3, )) ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} status = manager.status() for name, info in status.items(): print(f"{name}: {info['status']}") ``` *** ## How failover activates during retries Failover now drives LLM retries through direct integration with the retry mechanism: ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant LLM participant FailoverManager participant Provider A participant Provider B User->>Agent: Request Agent->>LLM: Call LLM->>FailoverManager: get_next_profile() FailoverManager-->>LLM: Provider A profile LLM->>LLM: Apply profile (api_key, base_url, model) LLM->>Provider A: API call Provider A--xLLM: Rate limit error LLM->>FailoverManager: mark_failure(profile, error, is_rate_limit=True) LLM->>FailoverManager: get_next_profile() FailoverManager-->>LLM: Provider B profile LLM->>LLM: Switch api_key, base_url, model LLM->>Provider B: Retry call Provider B-->>LLM: Success LLM->>FailoverManager: mark_success(profile) LLM-->>Agent: Response Agent-->>User: Final response ``` * On every LLM call, the system first gets the current profile via `get_next_profile()` and applies its `api_key`, `base_url`, and `model` settings * On success, `mark_success(profile)` is called to track the working provider * On failure, `mark_failure(profile, error, is_rate_limit=...)` marks the provider as failed, then `get_next_profile()` fetches the next available provider * Profile switching **overrides** non-retryable classificationβ€”one extra attempt is always granted after switching providers * The LLM automatically updates request parameters (api\_key, base\_url, model) when switching between profiles *** ## How It Works ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Agent participant Manager participant Primary participant Secondary Agent->>Manager: Request Manager->>Primary: Try Primary Primary--xManager: Rate Limited Manager->>Manager: Wait + Backoff Manager->>Secondary: Try Secondary Secondary-->>Manager: Success Manager-->>Agent: Response ``` | Component | Role | | ------------------- | --------------------------------- | | **AuthProfile** | Credentials for a single provider | | **FailoverManager** | Orchestrates failover logic | | **FailoverConfig** | Retry and backoff settings | | **ProviderStatus** | Tracks provider health | *** ## Configuration Options Manager class reference Provider credential profile ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import FailoverConfig config = FailoverConfig( max_retries=3, retry_delay=1.0, exponential_backoff=True, max_retry_delay=60.0, ) ``` | Option | Type | Default | Description | | ------------------------ | ------- | ------- | ----------------------------- | | `max_retries` | `int` | `3` | Maximum retry attempts | | `retry_delay` | `float` | `1.0` | Initial retry delay | | `exponential_backoff` | `bool` | `True` | Use exponential backoff | | `max_retry_delay` | `float` | `60.0` | Maximum retry delay | | `cooldown_on_rate_limit` | `float` | `60.0` | Rate limit cooldown (seconds) | | `cooldown_on_error` | `float` | `30.0` | Error cooldown (seconds) | | `rotate_on_success` | `bool` | `False` | Rotate profiles on success | *** ## Auth Profiles Configure credentials for each provider: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import AuthProfile profile = AuthProfile( name="openai-primary", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=1, rate_limit_rpm=100, ) ``` | Field | Type | Description | | ---------------- | ------ | ------------------------------------------- | | `name` | `str` | Unique profile identifier | | `provider` | `str` | Provider: openai, anthropic, etc. | | `api_key` | `str` | API key (masked in logs) | | `base_url` | `str` | Custom API endpoint | | `model` | `str` | Default model for this profile | | `priority` | `int` | Failover priority (lower = higher priority) | | `rate_limit_rpm` | `int` | Requests per minute limit | | `rate_limit_tpm` | `int` | Tokens per minute limit | | `metadata` | `dict` | Additional provider-specific config | *** ## Common Patterns ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import AuthProfile, FailoverManager manager = FailoverManager() # Add multiple providers manager.add_profile(AuthProfile( name="openai", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=1, )) manager.add_profile(AuthProfile( name="anthropic", provider="anthropic", api_key=os.getenv("ANTHROPIC_API_KEY"), priority=2, )) manager.add_profile(AuthProfile( name="groq", provider="groq", api_key=os.getenv("GROQ_API_KEY"), priority=3, )) ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import AuthProfile, FailoverManager manager = FailoverManager() manager.add_profile(AuthProfile( name="gpt-4o-mini", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=1, model="gpt-4o-mini", )) manager.add_profile(AuthProfile( name="gpt-4o", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), priority=2, model="gpt-4o", )) ``` ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import os from praisonaiagents import AuthProfile, FailoverManager manager = FailoverManager() manager.add_profile(AuthProfile( name="openai-us", provider="openai", api_key=os.getenv("OPENAI_API_KEY"), base_url="https://api.openai.com/v1", priority=1, )) manager.add_profile(AuthProfile( name="azure-eu", provider="azure", api_key=os.getenv("AZURE_OPENAI_API_KEY"), base_url=os.getenv("AZURE_OPENAI_ENDPOINT"), priority=2, )) ``` *** ## Failover Callbacks React to failover events: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import FailoverManager, FailoverConfig def on_failover(from_profile, to_profile, error): print(f"Failing over from {from_profile} to {to_profile}") print(f"Reason: {error}") # Log to monitoring system config = FailoverConfig( on_failover=on_failover ) manager = FailoverManager(config) ``` **Sync + async parity (from PraisonAI #2386).** The `retry` callback now fires on **both** the sync (`agent.chat`) and async (`await agent.achat`) LLM paths. Older versions silently skipped the callback on async calls β€” upgrade to get retry observability for gateway bots, async tools, and any `await`-based agent code. ### Failover callbacks run outside the pool lock Your `on_failover` callbacks are invoked **after** `FailoverManager` releases its internal lock (PraisonAI PR #3574). This means: * Blocking work in a callback (network I/O, logging, metrics) does **not** stall other agents queuing on the pool. * A callback that acquires an external lock will not deadlock against the pool's lock. * Callbacks still run in the thread that triggered the failover β€” if you care about ordering, that is preserved. *** ## Provider Status Monitor provider health: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import FailoverManager manager = FailoverManager() # Get status of all providers status = manager.status() for name, info in status.items(): print(f"{name}: {info['status']}") print(f" Failures: {info['failure_count']}") print(f" Last success: {info['last_success']}") # Reset a provider after recovery manager.mark_success("openai") # Reset all profiles manager.reset_all() ``` *** ## Thread safety `FailoverManager` is safe to **share across concurrent agents and threads** β€” it is the process-wide credential pool (PraisonAI PR #3574). Concurrent agents hitting a rate limit at the same time cannot observe torn reads or corrupt the pool: every read and mutation of the profile list and each profile's status is serialised internally. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR A1[πŸ€– Agent 1] --> Pool{πŸ”’ FailoverManager} A2[πŸ€– Agent 2] --> Pool A3[πŸ€– Agent 3] --> Pool Pool --> Profiles[πŸ—‚οΈ Shared Profiles] classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff classDef pool fill:#F59E0B,stroke:#7C90A0,color:#fff classDef store fill:#189AB4,stroke:#7C90A0,color:#fff class A1,A2,A3 agent class Pool pool class Profiles store ``` Wire one `FailoverManager` into many agents and let them share the same credential pool β€” no external locking required. See [Thread Safety](/docs/features/thread-safety) for how this fits with the rest of the SDK's concurrency model. *** ## Cross-Replica Coordination `FailoverManager` accepts an optional `coordinator=` argument (implementing `QuotaCoordinatorProtocol`) that shares credential cooldowns across replicas. When omitted, a private in-memory coordinator is used and behaviour is exactly as before. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import FailoverManager from praisonaiagents.llm import LocalQuotaCoordinator # Share cooldowns across managers in the same process. shared = LocalQuotaCoordinator() manager = FailoverManager(coordinator=shared) ``` Benches key on `AuthProfile.credential_id` β€” a stable, non-secret hash of `provider + base_url + api_key` β€” so two profiles named `"default"` with different keys never collide. See [Quota Coordinator](/docs/features/quota-coordinator). *** ## Best Practices Always have at least 2-3 providers configured. This ensures availability even during major outages. Enable `exponential_backoff=True` to avoid hammering providers during issues. This helps you stay within rate limits. Order providers by cost and reliability. Put cheaper/faster providers first, with premium providers as fallback. Use the `on_failover` callback to track when failovers occur. This helps identify provider issues early. The callback fires on both sync and async LLM paths β€” works the same whether your agent runs via `agent.chat()` or `await agent.achat()`. Pair failover with [LLM Error Classification](/docs/features/llm-error-classification) so `FailoverDecision` coordinates profile rotation with typed errors. Load keys from environment variables or a secrets manager β€” never commit credentials to version control. *** ## Related Share credential cooldowns across replicas Typed errors that drive failover decisions Supported LLM providers