Real API Testing CLI

CLI commands for running integration tests with real API keys

Real API Testing CLI

Commands for running integration tests with real API keys.

Commands

Run All Real API Tests

export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py -v

Output:

tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_simple_chat PASSED
tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_with_tool PASSED
tests/integration/test_real_api.py::TestAgentRealAPI::test_agent_chat_history PASSED
tests/integration/test_real_api.py::TestLiteAgentRealAPI::test_lite_agent_with_openai PASSED
tests/integration/test_real_api.py::TestLiteAgentRealAPI::test_lite_agent_no_litellm_loaded PASSED

Run Specific Provider Tests

OpenAI Tests

export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestAgentRealAPI -v

Anthropic Tests

export ANTHROPIC_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestAnthropicAPI -v

Google Gemini Tests

export GOOGLE_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestGoogleAPI -v

Run LiteAgent Tests

export OPENAI_API_KEY="your-key"
export RUN_REAL_KEY_TESTS=1
python -m pytest tests/integration/test_real_api.py::TestLiteAgentRealAPI -v

Quick Verification

Verify Agent Works

export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents import Agent

agent = Agent(
    name='Test',
    instructions='Reply with one word.',
    llm='gpt-4o-mini',
    output="silent"
)

response = agent.chat('Say hello')
print(f'Response: {response}')
print('Agent test: OK')
"

Verify LiteAgent Works

export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents.lite import LiteAgent, create_openai_llm_fn

llm_fn = create_openai_llm_fn(model='gpt-4o-mini')
agent = LiteAgent(name='Test', llm_fn=llm_fn)

response = agent.chat('Say hi')
print(f'Response: {response}')
print('LiteAgent test: OK')
"

Verify Tool Execution

export OPENAI_API_KEY="your-key"
python -c "
from praisonaiagents import Agent

def add(a: int, b: int) -> int:
    '''Add two numbers.'''
    return a + b

agent = Agent(
    name='Math',
    instructions='Use tools.',
    llm='gpt-4o-mini',
    tools=[add],
    output="silent"
)

response = agent.chat('What is 5+3?')
assert '8' in response, f'Expected 8 in response: {response}'
print('Tool test: OK')
"

Environment Variables

Variable	Description	Required
RUN_REAL_KEY_TESTS	Enable real API tests	Yes
OPENAI_API_KEY	OpenAI API key	For OpenAI tests
ANTHROPIC_API_KEY	Anthropic API key	For Anthropic tests
GOOGLE_API_KEY	Google API key	For Gemini tests

CI/CD Integration

GitHub Actions

name: Real API Tests

on:
  workflow_dispatch:

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e ".[test]"
      - name: Run Tests
        env:
          RUN_REAL_KEY_TESTS: "1"
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python -m pytest tests/integration/test_real_api.py -v

Shell Script

#!/bin/bash
set -e

export RUN_REAL_KEY_TESTS=1

if [ -z "$OPENAI_API_KEY" ]; then
    echo "Error: OPENAI_API_KEY not set"
    exit 1
fi

python -m pytest tests/integration/test_real_api.py -v
echo "All real API tests passed!"

Troubleshooting

Tests Skipped

If tests are skipped, ensure environment variable is set:

# Check if variable is set
echo $RUN_REAL_KEY_TESTS

# Set it
export RUN_REAL_KEY_TESTS=1

API Key Errors

# Verify key is set
python -c "import os; print('Key set:', bool(os.environ.get('OPENAI_API_KEY')))"

Rate Limits

If hitting rate limits, add delays between tests:

python -m pytest tests/integration/test_real_api.py -v --tb=short -x

Related

• Real API Testing (Code)
• Performance CLI
• Evaluation CLI