Advanced Multi-Provider Patterns

Advanced patterns for multi-provider LLM switching including fallback, load balancing, and circuit breakers

While basic multi-provider support lets you assign different LLMs to different agents, ModelRouter and RouterAgent add dynamic switching, cost optimisation, and resilience.

from praisonaiagents import RouterAgent

agent = RouterAgent(name="router", instructions="Route tasks to the best provider.")
agent.start("Summarise this report using the cheapest capable model.")

The user submits a task; the router picks a provider based on policy, cost, and availability.

How It Works

PR #2122: A model string like my-custom-model no longer defaults to the OpenAI provider — it raises ValueError. Use the provider/model form, e.g. ollama/llama3, bedrock/anthropic.claude-3-sonnet.

Non-built-in prefixes (bedrock/, deepseek/, any custom name) work when either LiteLLM natively supports them or you have registered a provider for them via register_llm_provider(...). The wrapper resolves registered providers before falling through to the LiteLLM ladder (PR #3011).

Model string format

Use either a recognised prefix (gpt-, claude-, gemini-) or explicit provider/model form:

Form	Example
Prefix	gpt-4o, claude-3-5-sonnet
Provider/model	ollama/llama3, bedrock/anthropic.claude-3-sonnet, deepseek/deepseek-chat

See Fail-Loud Defaults for migration guidance.

Quick Start

1. Simple Usage

   from praisonaiagents import RouterAgent
   
   router_agent = RouterAgent(
       models=["gpt-4o-mini", "gemini/gemini-1.5-flash"],
       routing_strategy="auto",
   )
   
   response = router_agent.run("What's the weather today?")
   
2. With Configuration

   from praisonaiagents import RouterAgent, Agent
   
   router = RouterAgent(
       models=[
           "gpt-4o",
           "gpt-4o-mini",
           "gemini/gemini-1.5-flash",
           "deepseek/deepseek-chat",
       ],
       routing_strategy="cost-optimized",
       fallback_model="gpt-4o-mini",
   )
   
   agent = Agent(name="Resilient Agent", router=router)
   response = agent.run("Analyze this complex financial report...")
   

Overview

The advanced multi-provider system enables:

• Dynamic model selection based on task requirements
• Automatic fallback when providers fail
• Cost-optimised routing for different task complexities
• Performance-based routing for critical operations
• Load balancing across providers
• Circuit breaker patterns for provider health

Routing Strategies

Automatic Routing (“auto”)

Analyses task complexity and requirements to select the best model:

router_agent = RouterAgent(
    models=["gpt-4o", "claude-3-opus-20240229", "gemini/gemini-1.5-pro"],
    routing_strategy="auto"
)

Cost-Optimized Routing

Prioritises cheaper models while ensuring task completion:

router_agent = RouterAgent(
    models=["gpt-4o", "gpt-4o-mini", "gemini/gemini-1.5-flash"],
    routing_strategy="cost-optimized",
    cost_threshold=0.005
)

usage = router_agent.get_usage_summary()
print(f"Total cost: ${usage['total_cost']:.4f}")

Performance-Optimized Routing

Prioritises capability and reliability for critical tasks:

router_agent = RouterAgent(
    models=["gpt-4o", "claude-3-opus-20240229"],
    routing_strategy="performance-optimized"
)

Advanced Patterns

Fallback Mechanism

from praisonaiagents import RouterAgent, Agent

router = RouterAgent(
    models=["gpt-4o", "claude-3-haiku-20240307", "gemini/gemini-1.5-flash"],
    fallback_model="gpt-4o-mini",
    routing_strategy="auto"
)

agent = Agent(name="Resilient Agent", router=router)

Task-Based Routing

from praisonaiagents import ModelRouter, RouterAgent

class CustomRouter(ModelRouter):
    def select_model(self, task_description: str, **kwargs):
        if any(k in task_description.lower() for k in ["code", "programming", "function"]):
            return "deepseek/deepseek-coder"
        if any(k in task_description.lower() for k in ["creative", "story", "poem"]):
            return "claude-3-opus-20240229"
        return "gemini/gemini-1.5-flash"

router_agent = RouterAgent(router=CustomRouter())

Provider Health Monitoring

from praisonaiagents import RouterAgent

class MonitoredRouter(RouterAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.provider_stats = {}

    def track_performance(self, model, latency, success):
        if model not in self.provider_stats:
            self.provider_stats[model] = {"total_calls": 0, "failures": 0, "avg_latency": 0}
        stats = self.provider_stats[model]
        stats["total_calls"] += 1
        if not success:
            stats["failures"] += 1

router = MonitoredRouter(
    models=["gpt-4o", "claude-3-haiku-20240307", "gemini/gemini-1.5-flash"]
)

Load Balancing

from praisonaiagents import RouterAgent
import random

class LoadBalancedRouter(RouterAgent):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.model_usage = {model: 0 for model in self.models}

    def select_model_balanced(self):
        min_usage = min(self.model_usage.values())
        candidates = [m for m, u in self.model_usage.items() if u == min_usage]
        selected = random.choice(candidates)
        self.model_usage[selected] += 1
        return selected

router = LoadBalancedRouter(
    models=["gpt-4o-mini", "gemini/gemini-1.5-flash", "claude-3-haiku"]
)

Circuit Breaker Pattern

from praisonaiagents import RouterAgent
import time

class CircuitBreakerRouter(RouterAgent):
    def __init__(self, *args, failure_threshold=5, timeout=300, **kwargs):
        super().__init__(*args, **kwargs)
        self.failure_threshold = failure_threshold
        self.timeout = timeout
        self.failures = {}
        self.circuit_open = {}

    def is_available(self, model):
        if model in self.circuit_open:
            if time.time() - self.circuit_open[model] > self.timeout:
                del self.circuit_open[model]
                self.failures[model] = 0
                return True
            return False
        return True

router = CircuitBreakerRouter(
    models=["gpt-4o", "claude-3-haiku-20240307", "gemini/gemini-1.5-flash"]
)

Integration with AutoAgents

from praisonaiagents import AutoAgentTeam, RouterAgent

router = RouterAgent(
    models=["gpt-4o", "claude-3-haiku-20240307", "gemini/gemini-1.5-flash"],
    routing_strategy="cost-optimized"
)

auto_agents = AutoAgentTeam(instructions="Create a research team", router=router)
agents = auto_agents.create_agents()

Best Practices

Configure accurate model profiles

Pass model_profiles with cost, strengths, and context window so routing decisions reflect real capabilities.

Monitor and cap costs

Use get_usage_summary() and set cost_threshold or daily budgets before production traffic.

Always set a fallback model

Provide fallback_model so a single provider outage does not halt the agent.

Use explicit provider/model strings

Prefer ollama/llama3 over ambiguous bare model names to avoid provider inference errors.

Related

Memory Management

Stateful multi-provider agents with persistent memory.

Fail-Loud Defaults

Provider inference rules and migration guidance.