Benchmarks

Performance benchmarks comparing PraisonAI Agents with other frameworks

​

Performance Benchmarks

PraisonAI Agents includes comprehensive benchmarks to measure and compare performance against other popular AI agent frameworks.

​

Benchmark Types

Benchmark	Description	File
Simple	Agent instantiation time (no API calls, no tools)	simple_benchmark.py
Tools	Agent instantiation with tools	tools_benchmark.py
Execution	Real agent execution with LLM API calls	execution_benchmark.py

​

Quick Start

cd src/praisonai-agents

# Run instantiation benchmark
python benchmarks/simple_benchmark.py

# Run tools benchmark
python benchmarks/tools_benchmark.py

# Run real execution benchmark (requires API key)
export OPENAI_API_KEY=your_key
python benchmarks/execution_benchmark.py

​

Simple Benchmark

Measures PraisonAI agent instantiation time (no API calls, no tools) and compares against installed frameworks. Every framework is built without tools.

​

CLI Options

python benchmarks/simple_benchmark.py [OPTIONS]

Option	Default	Description
--save	False	Save SIMPLE_BENCHMARK_RESULTS.md. Does not touch BENCHMARK_RESULTS.md or README.md.
--skip-crewai	False	Skip the optional CrewAI benchmark (its import can hang 4+ min on some platforms).
--json	False	Emit {"results": {...}, "iterations": N, "warmup": M} to stdout. Human progress output is suppressed; save-status goes to stderr so stdout stays machine-readable.
--fail-over-threshold	False	Exit non-zero when PraisonAI instantiation exceeds praisonai_instantiation_us_max in thresholds.json. Prints ERROR: PraisonAI instantiation X us > threshold Y us on stderr.

​

Thresholds file

The script reads benchmarks/thresholds.json for CI budgets:

{
  "praisonai_instantiation_us_max": 150,
  "iterations": 100,
  "warmup": 10
}

• praisonai_instantiation_us_max — hard cap (µs) for --fail-over-threshold. Set to ≈3× the typical ~48 μs we measure on the GitHub-hosted runner.
• iterations / warmup — override the script’s defaults.

​

Examples

# Default run (same output as before)
python benchmarks/simple_benchmark.py

# CI baseline: skip the hang-prone framework, emit machine-readable JSON
python benchmarks/simple_benchmark.py --skip-crewai --json

# CI gate: fail the job on a regression
python benchmarks/simple_benchmark.py --skip-crewai --json --fail-over-threshold \
  | tee agent-perf-baseline.json

​

Performance Benchmark

performance_benchmark.py produces the cross-framework table in BENCHMARK_RESULTS.md. Competitor frameworks are imported lazily, so it runs with only praisonaiagents installed and skips anything missing.

Option	Default	Description
--save	False	Save results to BENCHMARK_RESULTS.md.
--iterations	1000	Measured iterations per framework.
--warmup	10	Warmup runs per framework.
--only	all	Benchmark only the listed frameworks: praisonai, praisonai_litellm, agno, pydantic, openai_agents, langgraph, crewai.

# PraisonAI only, quick run
python benchmarks/performance_benchmark.py --only praisonai --iterations 200

​

Execution Benchmark

The execution benchmark tests real agent execution with actual LLM API calls.

Each framework makes untimed warm-up calls first, so cold imports and the first connection are not counted. The timed runs are interleaved round by round, so network drift affects every framework alike. A run that raises or returns an empty answer counts as a failure, not a timing. The summary reports the median, the mean and the number of successful and failed runs.

​

CLI Options

python benchmarks/execution_benchmark.py [OPTIONS]

Option	Short	Default	Description
--model	-m	gpt-4o-mini	Model to use
--iterations	-i	10	Timed rounds; each round runs every framework once
--warmup	-w	1	Untimed warm-up calls per framework
--prompt	-p	"What is 2+2?..."	Prompt to use
--save	-	False	Save results to EXECUTION_BENCHMARK_RESULTS.md

​

Examples

# Default (10 rounds after 1 warm-up call each, gpt-4o-mini, not saved)
python benchmarks/execution_benchmark.py

# More rounds for steadier medians
python benchmarks/execution_benchmark.py -i 30

# Custom model
python benchmarks/execution_benchmark.py -m gpt-4o

# Custom prompt
python benchmarks/execution_benchmark.py --prompt "What is the capital of France?"

# Save results
python benchmarks/execution_benchmark.py --save

# Combine options
python benchmarks/execution_benchmark.py -m gpt-4o -i 20 --save

​

Results

​

Agent Instantiation (October 2026)

Framework	Avg Time (μs)
Agno	9.5
PraisonAI 1.7.12	45.6
PraisonAI (LiteLLM route)	46.9
OpenAI Agents SDK	591
PydanticAI	1,022
LangGraph	3,233

Measured 2026-10-15 on Linux x86_64 (4 CPUs, Python 3.12) · one tool · 10 warmup + 1000 iterations · single script run with the same setup for every framework · source: src/praisonai-agents/benchmarks/BENCHMARK_RESULTS.md

CrewAI is not in this run — its pip resolve hangs. All other rows share the same setup; the previous mislabelled “1.00x (fastest)” row in performance_benchmark.py was fixed in PR #5836, so ratios now compute against the row that is actually fastest.

Historical (superseded)

June 2026 (praisonaiagents 1.6.60): ~13.6 μs avg, measured on a different machine and setup. A same-hardware re-measurement on 2026-10-26 put 1.6.60 at 20.65 μs and main (1.7.11) at 47.92 μs avg. Earlier results (December 2025) and the tools and execution benchmarks were measured under different setups and are no longer published. Past results live in benchmarks/archive/.

​

Frameworks Compared

Framework	Execution Method
PraisonAI	agent.start()
PraisonAI (LiteLLM)	agent.start()
Agno	agent.run()
CrewAI	crew.kickoff()
OpenAI Agents SDK	Runner.run()
LangGraph	create_react_agent()
PydanticAI	Agent()

​

Output Files

Benchmarks save results to markdown files in the benchmarks/ directory:

File	Description
BENCHMARK_RESULTS.md	Current cross-framework results (performance_benchmark.py --save)
SIMPLE_BENCHMARK_RESULTS.md	simple_benchmark.py --save output
TOOLS_BENCHMARK_RESULTS.md	tools_benchmark.py --save output
EXECUTION_BENCHMARK_RESULTS.md	execution_benchmark.py --save output
archive/	Past results

Nothing is written unless you pass --save.

​

Methodology

Methodology

Measures only Agent(...) construction: no LLM call, no import time. 10 warmup + 1000 iterations with time.perf_counter(), one tool. Single machine; numbers vary by hardware and version. Reproduce with python benchmarks/performance_benchmark.py — competitor imports are lazy, so it runs with only praisonaiagents installed. Last measured: 2026-10-15.

​

Running Your Own Benchmarks

1. Clone the repository:

git clone https://github.com/MervinPraison/PraisonAI.git
cd PraisonAI/src/praisonai-agents

2. Install dependencies:

pip install "praisonaiagents[llm]" agno crewai pydantic-ai openai-agents langgraph langchain-openai

[llm] installs LiteLLM, which the “PraisonAI (LiteLLM)” rows need. The LangGraph row needs langchain-openai. A script skips any framework that isn’t installed, without an error.

3. Set your API key:

export OPENAI_API_KEY=your_key

4. Run benchmarks:

python benchmarks/simple_benchmark.py
python benchmarks/tools_benchmark.py
python benchmarks/execution_benchmark.py