Real API Key Testing

Gated integration tests with real API keys

Run integration tests against live LLM providers — gated by RUN_REAL_KEY_TESTS so CI stays free and fast by default.

For offline tests that never reach a provider — no API keys, no network, no bill — see Offline Testing.

import os
import pytest
from praisonaiagents import Agent

@pytest.mark.skipif(
    os.getenv("RUN_REAL_KEY_TESTS", "").lower() not in ("1", "true", "yes"),
    reason="Set RUN_REAL_KEY_TESTS=1 to enable",
)
def test_agent_simple_chat():
    agent = Agent(
        name="TestAgent",
        instructions="Reply briefly.",
        llm="gpt-4o-mini",
    )
    response = agent.start("Say hello in one word")
    assert len(response) > 0

The user sets RUN_REAL_KEY_TESTS=1; pytest runs a real agent chat against live provider keys.

How It Works

Quick Start

1. Simple Usage

   Enable the gate and set your API key:

   export RUN_REAL_KEY_TESTS=1
   export OPENAI_API_KEY="$(printenv OPENAI_API_KEY)"
   
   python -m pytest tests/integration/test_real_api.py -v
   
2. With Configuration

   Run a single test class for one provider:

   export RUN_REAL_KEY_TESTS=1
   export ANTHROPIC_API_KEY="$(printenv ANTHROPIC_API_KEY)"
   
   python -m pytest tests/integration/test_real_api.py::TestAnthropicAPI -v
   

---

How It Works

All tests in tests/integration/test_real_api.py use a module-level skip when RUN_REAL_KEY_TESTS is unset. Accepted values: 1, true, yes (case-insensitive).

Variable	Required	Purpose
RUN_REAL_KEY_TESTS	Yes	Enables live API tests
OPENAI_API_KEY	For OpenAI tests	Provider authentication
ANTHROPIC_API_KEY	For Claude tests	Optional provider
GOOGLE_API_KEY	For Gemini tests	Optional provider

---

Configuration Options

Option	Type	Default	Description
RUN_REAL_KEY_TESTS	env	unset	Master gate — tests skipped when absent
llm	str	"gpt-4o-mini"	Use fast models to minimise cost
output	preset	"silent"	Suppress Rich output in tests

---

Best Practices

Keep prompts short

Use minimal prompts and prefer gpt-4o-mini — real tests consume tokens and cost money.

Never commit API keys

Read keys from environment variables; store CI secrets in GitHub Secrets.

Gate CI with workflow_dispatch

Trigger real API tests manually in GitHub Actions, not on every push.

Use the shared skip decorator

Reuse pytest.mark.skipif on RUN_REAL_KEY_TESTS for consistent gating across modules.

---

CI Example

name: Real API Tests
on:
  workflow_dispatch:
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install praisonaiagents pytest
      - name: Run real API tests
        env:
          RUN_REAL_KEY_TESTS: "1"
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: python -m pytest tests/integration/test_real_api.py -v

---

Related

Offline Testing

Unit-test agents without hitting a provider

Real API Testing CLI

CLI commands for live API tests

Performance Benchmarks

Measure agent performance