LLM Configuration

This page provides comprehensive documentation for configuring Large Language Models (LLMs) in PraisonAI, including retry mechanisms, timeout settings, custom headers, and advanced optimization options.

Environment Variable Precedence

PraisonAI resolves LLM configuration from environment variables using a documented precedence order, ensuring consistent behavior across all components.

Variable	Purpose	Precedence
`MODEL_NAME`	Model name (highest priority)	1
`OPENAI_MODEL_NAME`	Model name (legacy compat)	2
`OPENAI_BASE_URL`	LLM endpoint URL (highest priority)	1
`OPENAI_API_BASE`	LLM endpoint URL (legacy compat)	2
`OLLAMA_API_BASE`	Ollama endpoint URL	3
`ANTHROPIC_API_KEY`	Anthropic API key (for `anthropic/*` models)	—
`GOOGLE_API_KEY`	Google API key (for `google/*` models)	—
`GEMINI_API_KEY`	Gemini API key (for `gemini/*` models)	—
`GROQ_API_KEY`	Groq API key (for `groq/*` models)	—
`COHERE_API_KEY`	Cohere API key (for `cohere/*` models)	—
`OPENROUTER_API_KEY`	OpenRouter API key (for `openrouter/*` models)	—
`OLLAMA_API_KEY`	Ollama API key (for `ollama/*` models)	—
`OPENAI_API_KEY`	OpenAI API key (for OpenAI models and fallback)	—

Defaults:

Model: gpt-4o-mini
Base URL: Provider-specific or https://api.openai.com/v1
API Key: None

PraisonAI resolves these once at startup; all internal components use the same values. For complete environment variable configuration guide, see LLM Endpoint Configuration.

Default LLM Timeouts

PraisonAI bounds LLM calls at two layers so an unresponsive provider cannot pin a request indefinitely. See LLM Timeouts for the full guide.

Default on direct LLM / Agent(llm=“model”) path

Agents built with praisonaiagents get a 600 second (10 minute) per-request timeout when timeout is not set.

You write	Resolved timeout	Sent to provider
`llm="gpt-4o"` or `llm={"model": "gpt-4o"}`	`600`	`timeout=600`
`llm={"model": "gpt-4o", "timeout": None}`	`None`	Not sent — litellm’s ~100 min ceiling applies
`llm={"model": "gpt-4o", "timeout": 0}`	`0`	Not sent — litellm’s ~100 min ceiling applies
`llm={"model": "gpt-4o", "timeout": 30}`	`30`	`timeout=30`

from praisonaiagents import Agent

agent = Agent(name="Assistant", llm={"model": "gpt-4o", "timeout": 30})  # 30s wins over 600s

For streaming calls the 600 s is a read-gap bound (max gap between chunks), not the total call duration.

Default on framework registry / gateway path

Calls dispatched through the praisonai provider registry (openai, anthropic, google, groq, mistral, bedrock, azure, ollama, xai, deepseek, cohere, together, sagemaker) and gateway providers (openrouter, litellm-proxy, custom-gateway) are bounded by two defaults.

Default	Value	Overridable via
`timeout`	`60.0` seconds	Per-call `timeout=` kwarg (wins) or env var `PRAISONAI_LLM_TIMEOUT` (float seconds)
`num_retries`	`2`	Per-call `num_retries=` kwarg

# Tighten the registry/gateway default for the whole process
export PRAISONAI_LLM_TIMEOUT=10

A malformed value (PRAISONAI_LLM_TIMEOUT=abc) logs a warning and falls back to 60.

Behaviour change (PraisonAI PR #5972): an unset timeout on the direct LLM / Agent(llm=...) path now resolves to 600 s instead of being left to litellm’s 6000 s ceiling — a stuck provider now stops the turn after 10 minutes instead of 100. Pass timeout=None to restore the old behaviour.The registry/gateway defaults were introduced earlier in v4.6.163 (PraisonAI PR #3879). Before that, registry calls had no timeout — a black-holed provider would hold a request open until the OS TCP keepalive gave up (~2 hours on Linux defaults).

Core LLM Configuration

Basic Setup

from praisonaiagents import Agent

agent = Agent(
    name="Assistant",
    llm="gpt-4o",
    llm={
        "temperature": 0.7,
        "max_tokens": 4000,
        "timeout": 60,
        "api_key": "your-api-key"
    }
)

Behaviour change (PraisonAI PR #4245). Before this release, temperature set inside Agent(llm={...}) was silently overridden by a hardcoded 1.0 at request time — configured values never reached the provider. It now reaches the request as written. If you set temperature via the llm dict, your output will change; if you did not set it, the provider default now applies (previously an explicit 1.0 was sent). Setting temperature per call still wins over the instance/config value.

Temperature precedence: per-call temperature= kwarg > Agent(llm={'temperature': ...}) / Agent(temperature=...) > provider default (temperature omitted from the request). When temperature is not configured anywhere, the parameter is omitted from the request rather than sent as an explicit None or 1.0. This matters for providers whose default differs from 1.0 — for example reasoning models that reject non-default temperatures (see the note in JS output).

To confirm the value reached the request, enable LiteLLM debug logging or check your provider dashboard for the temperature recorded on the call.

Provider-Specific Configuration

# OpenAI Configuration
openai_config = {
    "model": "gpt-4o",
    "api_key": "sk-...",
    "organization": "org-...",
    "base_url": "https://api.openai.com/v1",
    "timeout": 60,
    "max_retries": 3,
    "temperature": 0.7,
    "max_tokens": 4000,
    "presence_penalty": 0.1,
    "frequency_penalty": 0.1
}

# Anthropic Configuration
anthropic_config = {
    "model": "claude-3-sonnet-20240229",
    "api_key": "sk-ant-...",
    "base_url": "https://api.anthropic.com",
    "timeout": 90,
    "max_retries": 3,
    "temperature": 0.7,
    "max_tokens": 4000,
    "anthropic_version": "2023-06-01"
}

# Custom/Local LLM Configuration
custom_config = {
    "model": "custom-model",
    "base_url": "http://localhost:8000",
    "timeout": 120,
    "headers": {
        "Authorization": "Bearer custom-token"
    }
}

Retry Logic Configuration

Basic Retry Settings

retry_config = {
    "max_retries": 3,
    "retry_delay": 2.0,  # seconds
    "retry_multiplier": 2.0,  # exponential backoff multiplier
    "max_retry_delay": 30.0,  # maximum delay between retries
    "retry_on_status": [429, 500, 502, 503, 504],  # HTTP status codes
    "retry_on_errors": [
        "RateLimitError",
        "APIConnectionError",
        "Timeout",
        "ServiceUnavailableError"
    ]
}

Advanced Retry Logic

advanced_retry_config = {
    "retry_strategy": "exponential_backoff_with_jitter",
    "max_retries": 5,
    "base_delay": 1.0,
    "max_delay": 60.0,
    "jitter": 0.1,  # 10% randomization
    
    # Error-specific retry behavior
    "error_retry_config": {
        "RateLimitError": {
            "max_retries": 10,
            "base_delay": 5.0,
            "respect_retry_after": True
        },
        "APIConnectionError": {
            "max_retries": 3,
            "base_delay": 2.0,
            "increase_timeout": True
        },
        "InsufficientQuotaError": {
            "max_retries": 0,  # Don't retry
            "fallback_model": "gpt-3.5-turbo"
        }
    },
    
    # Circuit breaker configuration (for custom integrations)
    # Note: Tool circuit breakers are automatic - see /features/tool-circuit-breaker
    "circuit_breaker": {
        "failure_threshold": 5,
        "recovery_timeout": 60.0,
        "success_threshold": 2,
        "timeout": 30.0,
        "graceful_degradation": True
    }
}

Custom Retry Logic Implementation

def custom_retry_handler(error, attempt, config):
    """Custom retry logic for specific scenarios"""
    if isinstance(error, RateLimitError):
        # Extract retry-after header if available
        retry_after = error.response.headers.get('retry-after', 60)
        return min(retry_after, config['max_delay'])
    
    elif isinstance(error, ModelOverloadedError):
        # Switch to a different model
        config['fallback_model'] = "gpt-3.5-turbo"
        return config['base_delay'] * (2 ** attempt)
    
    else:
        # Default exponential backoff
        return min(
            config['base_delay'] * (config['retry_multiplier'] ** attempt),
            config['max_delay']
        )

llm_config = {
    "retry_handler": custom_retry_handler,
    "max_retries": 5
}

Timeout Configuration

Timeout Settings

timeout_config = {
    # Basic timeout
    "timeout": 60,  # seconds
    
    # Detailed timeout configuration
    "timeout_config": {
        "connect": 5.0,  # Connection timeout
        "read": 60.0,    # Read timeout
        "write": 10.0,   # Write timeout
        "pool": 5.0      # Connection pool timeout
    },
    
    # Dynamic timeout based on request
    "dynamic_timeout": {
        "base": 30,
        "per_token": 0.01,  # Additional time per token
        "min": 10,
        "max": 300
    },
    
    # Timeout retry behavior
    "timeout_retry": {
        "increase_factor": 1.5,  # Increase timeout on retry
        "max_timeout": 300
    }
}

Request-Specific Timeouts

# Configure timeouts based on operation type
operation_timeouts = {
    "completion": {
        "timeout": 60,
        "dynamic": True,
        "factors": {
            "max_tokens": 0.01,
            "temperature": 1.2  # Higher temperature = more time
        }
    },
    "embedding": {
        "timeout": 30,
        "batch_factor": 0.1  # Per item in batch
    },
    "chat": {
        "timeout": 90,
        "message_factor": 5  # Per message in history
    }
}

Custom Headers Configuration

Basic Headers

headers_config = {
    "headers": {
        "Authorization": "Bearer your-api-key",
        "Content-Type": "application/json",
        "User-Agent": "PraisonAI/1.0",
        "X-Custom-Header": "custom-value"
    }
}

Dynamic Headers

import uuid

def generate_headers(request_type, model, **kwargs):
    """Generate headers dynamically based on request"""
    headers = {
        "User-Agent": f"PraisonAI/1.0 ({request_type})",
        "X-Model": model,
        "X-Request-ID": str(uuid.uuid4()),
        "X-Client-Version": "1.0.0"
    }
    
    # Add authentication
    if api_key := kwargs.get('api_key'):
        headers["Authorization"] = f"Bearer {api_key}"
    
    # Add custom headers for specific providers
    if "anthropic" in model:
        headers["anthropic-version"] = "2023-06-01"
    elif "openai" in model:
        headers["OpenAI-Beta"] = "assistants=v1"
    
    return headers

llm_config = {
    "headers_generator": generate_headers,
    "static_headers": {
        "X-Environment": "production"
    }
}

Provider-Specific Headers

import time

# OpenAI specific headers
openai_headers = {
    "OpenAI-Organization": "org-xxx",
    "OpenAI-Beta": "assistants=v1",
    "X-Request-ID": "unique-request-id"
}

# Anthropic specific headers
anthropic_headers = {
    "anthropic-version": "2023-06-01",
    "X-Request-Source": "praisonai"
}

# Custom authentication headers
custom_auth_headers = {
    "X-API-Key": "your-api-key",
    "X-API-Secret": "your-secret",
    "X-Timestamp": str(int(time.time())),
    "X-Signature": "generated-signature"
}

Advanced LLM Configuration

Load Balancing

load_balancing_config = {
    "strategy": "round_robin",  # or "least_latency", "weighted"
    "endpoints": [
        {
            "url": "https://api.openai.com/v1",
            "weight": 0.6,
            "models": ["gpt-4o", "gpt-3.5-turbo"]
        },
        {
            "url": "https://api.anthropic.com",
            "weight": 0.4,
            "models": ["claude-3-sonnet"]
        }
    ],
    "health_check": {
        "enabled": True,
        "interval": 60,
        "timeout": 5,
        "failure_threshold": 3
    }
}

Model Fallback Configuration

See Model Fallback for the full guide.

from praisonaiagents import Agent
from praisonaiagents.config import LLMConfig

agent = Agent(
    name="Assistant",
    llm=LLMConfig(
        model="gpt-4o",
        fallback_models=["claude-3-5-sonnet", "gpt-4o-mini"],
    ),
)

On transient errors (503, timeout, overloaded), the agent retries the same turn against the next model in the list. Configure fallbacks with LLMConfig(fallback_models=[...]) — not via raw config dict keys. This covers every agent shape — provider/model strings, custom base_url servers, and model-object LLMs — on sync, async, and streaming turns.

Request Optimization

optimization_config = {
    # Request batching
    "batching": {
        "enabled": True,
        "max_batch_size": 10,
        "batch_timeout": 0.1,  # seconds
        "dynamic_batching": True
    },
    
    # Response streaming
    "streaming": {
        "enabled": True,
        "chunk_size": 100,
        "buffer_size": 1000,
        "timeout_per_chunk": 30
    },
    
    # Caching
    "cache": {
        "enabled": True,
        "ttl": 3600,
        "max_size": 1000,
        "key_strategy": "semantic",  # or "exact"
        "similarity_threshold": 0.95
    },
    
    # Token optimization
    "token_optimization": {
        "compress_prompts": True,
        "remove_redundancy": True,
        "dynamic_max_tokens": True,
        "reserve_completion_tokens": 500
    }
}

Rate Limiting Configuration

rate_limit_config = {
    "rate_limits": {
        "requests_per_minute": 60,
        "tokens_per_minute": 90000,
        "requests_per_day": 10000
    },
    "rate_limit_strategy": "adaptive",  # or "fixed", "burst"
    "burst_config": {
        "burst_size": 10,
        "refill_rate": 1.0  # per second
    },
    "quota_management": {
        "track_usage": True,
        "warn_at_percentage": 80,
        "hard_limit_behavior": "queue"  # or "reject", "fallback"
    }
}

Complete Configuration Example

from praisonaiagents import Agent

# Comprehensive LLM configuration
agent = Agent(
    name="ProductionAgent",
    llm="gpt-4o",
    llm={
        # Model settings
        "temperature": 0.7,
        "max_tokens": 4000,
        "top_p": 0.9,
        "presence_penalty": 0.1,
        "frequency_penalty": 0.1,
        
        # Timeout configuration
        "timeout": 60,
        "timeout_config": {
            "connect": 5,
            "read": 60,
            "dynamic": True
        },
        
        # Retry configuration
        "max_retries": 5,
        "retry_delay": 2.0,
        "retry_multiplier": 2.0,
        "retry_on_status": [429, 500, 502, 503],
        
        # Headers
        "headers": {
            "User-Agent": "PraisonAI/1.0",
            "X-Request-Source": "production"
        },
        
        # Advanced features
        "streaming": True,
        "cache_enabled": True,
        
        # Rate limiting
        "rate_limit_config": {
            "requests_per_minute": 60,
            "adaptive": True
        }
    }
)

Model fallback is configured via LLMConfig(fallback_models=[...]), not keys inside the llm dict above. See Model Fallback.

Environment Variables

# Basic LLM settings
export OPENAI_API_KEY="sk-..."
export OPENAI_MODEL="gpt-4o"
export OPENAI_TEMPERATURE="0.7"

# Default LLM call timeout (seconds) — applied to every LiteLLM call
export PRAISONAI_LLM_TIMEOUT="60"

PRAISONAI_LLM_TIMEOUT is the only PRAISONAI_LLM_* variable wired into the SDK today — see Default LLM Timeouts.

Planned (not yet wired)

The variables below are not read by the SDK yet. Exporting them has no effect. They are listed for roadmap visibility only.

# Timeout siblings
export PRAISONAI_LLM_CONNECT_TIMEOUT="5"
export PRAISONAI_LLM_READ_TIMEOUT="60"

# Retry settings
export PRAISONAI_LLM_MAX_RETRIES="3"
export PRAISONAI_LLM_RETRY_DELAY="2"
export PRAISONAI_LLM_RETRY_MULTIPLIER="2"

# Headers
export PRAISONAI_LLM_USER_AGENT="PraisonAI/1.0"
export PRAISONAI_LLM_CUSTOM_HEADERS='{"X-Custom": "value"}'

# Advanced settings
export PRAISONAI_LLM_STREAMING="true"
export PRAISONAI_LLM_CACHE_ENABLED="true"
export PRAISONAI_LLM_RATE_LIMIT="60"

For a per-call retry override today, pass num_retries= on the call — the default is 2. For per-call header, timeout, or streaming control, set them inside the llm={...} dict on the Agent (see Core LLM Configuration).

Monitoring and Debugging

monitoring_config = {
    "logging": {
        "log_requests": True,
        "log_responses": True,
        "log_level": "INFO",
        "sanitize_keys": ["api_key", "authorization"]
    },
    "metrics": {
        "track_latency": True,
        "track_tokens": True,
        "track_costs": True,
        "export_interval": 60
    },
    "debugging": {
        "capture_raw_responses": False,
        "validate_responses": True,
        "break_on_error": False
    }
}

​LLM Configuration

​Environment Variable Precedence

​Default LLM Timeouts

​Default on direct LLM / Agent(llm=“model”) path

​Default on framework registry / gateway path

​Core LLM Configuration

​Basic Setup

​Provider-Specific Configuration

​Retry Logic Configuration

​Basic Retry Settings

​Advanced Retry Logic

​Custom Retry Logic Implementation

​Timeout Configuration

​Timeout Settings

​Request-Specific Timeouts

​Custom Headers Configuration

​Basic Headers

​Dynamic Headers

​Provider-Specific Headers

​Advanced LLM Configuration

​Load Balancing

​Model Fallback Configuration

​Request Optimization

​Rate Limiting Configuration

​Complete Configuration Example

​Environment Variables

​Planned (not yet wired)

​Monitoring and Debugging

​See Also

LLM Configuration

Environment Variable Precedence

Default LLM Timeouts

Default on direct LLM / Agent(llm=“model”) path

Default on framework registry / gateway path

Core LLM Configuration

Basic Setup

Provider-Specific Configuration

Retry Logic Configuration

Basic Retry Settings

Advanced Retry Logic

Custom Retry Logic Implementation

Timeout Configuration

Timeout Settings

Request-Specific Timeouts

Custom Headers Configuration

Basic Headers

Dynamic Headers

Provider-Specific Headers

Advanced LLM Configuration

Load Balancing

Model Fallback Configuration

Request Optimization

Rate Limiting Configuration

Complete Configuration Example

Environment Variables

Planned (not yet wired)

Monitoring and Debugging

See Also