RouteLLM Integration

Route requests between strong and weak models to optimize cost

from praisonaiagents import Agent

agent = Agent(
    llm="router-mf-0.5",
    base_url="http://localhost:6060/v1"  # RouteLLM server
)

response = agent.chat("What is 2+2?")
print(response)

Install RouteLLM

pip install routellm

Start RouteLLM Server

python -m routellm.openai_server \
  --routers mf \
  --strong-model gpt-4o \
  --weak-model gpt-4o-mini \
  --port 6060

Environment Variables

export OPENAI_API_KEY=your-api-key

Router Options

Router	Description	Command
mf	Matrix factorization (recommended)	--routers mf
sw_ranking	Similarity-weighted ranking	--routers sw_ranking
bert	BERT classifier	--routers bert
causal_llm	LLaMA classifier	--routers causal_llm
random	Random selection	--routers random

Threshold Configuration

The threshold (0.0-1.0) controls routing:

• 0.0: Always use weak model
• 1.0: Always use strong model
• 0.5: Balanced (default)

# More aggressive cost savings (use weak model more often)
agent = Agent(
    llm="router-mf-0.3",
    base_url="http://localhost:6060/v1"
)

# Higher quality (use strong model more often)
agent = Agent(
    llm="router-mf-0.7",
    base_url="http://localhost:6060/v1"
)

Multi-Agent Example

from praisonaiagents import Agent, PraisonAIAgentTeam, Task

# All agents use RouteLLM routing
researcher = Agent(
    name="Researcher",
    role="Research analyst",
    goal="Find information",
    llm="router-mf-0.5",
    base_url="http://localhost:6060/v1"
)

writer = Agent(
    name="Writer",
    role="Content writer",
    goal="Write content",
    llm="router-mf-0.5",
    base_url="http://localhost:6060/v1"
)

task1 = Task(
    description="Research AI trends",
    agent=researcher,
    expected_output="Research summary"
)

task2 = Task(
    description="Write article based on research",
    agent=writer,
    expected_output="Article"
)

agents = PraisonAIAgents(agents=[researcher, writer], tasks=[task1, task2])
result = agents.start()

Workflow Example

from praisonaiagents import Agent, AgentFlow

workflow = AgentFlow(
    steps=[
        Agent(
            name="Analyzer",
            role="Data analyst",
            llm="router-mf-0.5",
            base_url="http://localhost:6060/v1"
        ),
        Agent(
            name="Reporter",
            role="Report writer",
            llm="router-mf-0.5",
            base_url="http://localhost:6060/v1"
        )
    ]
)

result = workflow.run("Analyze sales data")

Server with Custom Models

# Use Claude as strong model
python -m routellm.openai_server \
  --routers mf \
  --strong-model anthropic/claude-3-5-sonnet-20241022 \
  --weak-model gpt-4o-mini \
  --port 6060

# Use Ollama models
python -m routellm.openai_server \
  --routers mf \
  --strong-model ollama/llama3.1:70b \
  --weak-model ollama/llama3.1:8b \
  --port 6060

Docker Deployment

docker run -d \
  -p 6060:6060 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  routellm/routellm \
  --routers mf \
  --strong-model gpt-4o \
  --weak-model gpt-4o-mini

Config File

Create config.yaml:

mf:
  checkpoint_path: "routellm/mf_gpt4_augmented"

Start with config:

python -m routellm.openai_server \
  --routers mf \
  --config config.yaml \
  --strong-model gpt-4o \
  --weak-model gpt-4o-mini \
  --port 6060

API Compatibility

RouteLLM server is OpenAI-compatible. Use with any OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:6060/v1",
    api_key="not-needed"  # Uses server's API key
)

response = client.chat.completions.create(
    model="router-mf-0.5",
    messages=[{"role": "user", "content": "Hello"}]
)

Cost Savings

RouteLLM can reduce costs by up to 85% while maintaining 95% quality on benchmarks.

Threshold	Cost Reduction	Quality Retention
0.3	~70%	~90%
0.5	~50%	~95%
0.7	~30%	~98%