Context Evaluator

Score multi-agent handoff fidelity and token-budget compliance without an LLM

Context Evaluator scores how much of one agent’s output reaches the next agent — plus whether each agent stayed inside its token budget — all without a live LLM call.

Quick Start

1. Score a 2-agent workflow

   from praisonaiagents import Agent, Task, PraisonAIAgents
   from praisonaiagents.eval import ContextEvaluator
   
   researcher = Agent(name="researcher", instructions="Research the topic")
   writer     = Agent(name="writer",     instructions="Write the final report")
   
   research = Task(description="Research AI agents", agent=researcher)
   report   = Task(description="Write the report", agent=writer)
   
   workflow = PraisonAIAgents(agents=[researcher, writer], tasks=[research, report])
   workflow.start()
   
   result = ContextEvaluator(
       trace_events=workflow.trace_events,
       agent_order=["researcher", "writer"],
   ).run(print_summary=True)
   
   print(result.overall_score)   # 0-10
   
2. Add a token-budget ledger

   from praisonaiagents.eval import ContextEvaluator
   
   result = ContextEvaluator(
       trace_events=workflow.trace_events,
       agent_order=["researcher", "writer"],
       budget_ledger=[
           {"agent_name": "researcher", "used_tokens": 8_200, "budget_tokens": 10_000},
           {"agent_name": "writer",     "used_tokens": 4_100, "budget_tokens": 6_000},
       ],
   ).run(print_summary=True)
   
   print(result.handoff_score)   # context flow (0-10)
   print(result.budget_score)    # budget compliance (0-10)
   
3. Run inside an EvalSuite

   from praisonaiagents.eval import ContextEvaluator, EvalSuite, AccuracyEvaluator
   
   suite = EvalSuite(evaluators=[
       ContextEvaluator(
           trace_events=workflow.trace_events,
           agent_order=["researcher", "writer"],
       ),
       AccuracyEvaluator(agent=writer, input_text="Summarize AI agents", expected_output="..."),
   ])
   report = suite.run()
   

---

How It Works

Context Evaluator reads the workflow trace and measures how much of each agent’s response tokens survive into the next agent’s request.

Signal	Source event	Score
Exact content match	llm_response → llm_request	10/10
Partial overlap	token intersection	min(10, overlap × 12 + 2)
Below 5	low overlap	flags content_loss_detected=True
Budget met	used <= budget	10/10
Over budget	overrun	max(1, 10 - overrun × 10)
No budget (0)	neutral	5.0

---

When to choose which dimension

Pick the dimensions that match the data you have.

---

Configuration Options

Full parameter narrative lives in the SDK source.

Option	Type	Default	Description
trace_events	list[Any]	None	Trace events, each with event_type / agent_name / data (objects or dicts)
agent_order	list[str]	None	Ordered agent names for the handoff chain (needs ≥ 2 for handoff scoring)
budget_ledger	list[dict]	None	List of {"agent_name", "used_tokens", "budget_tokens"} dicts
name	str	None	Evaluation name
save_results_path	str	None	Path to persist result JSON
verbose	bool	False	Enable verbose logging

Methods

Method	Returns	Purpose
evaluate_handoff()	list[ContextHandoffResult]	Token overlap for each adjacent agent pair
evaluate_budget()	list[BudgetComplianceResult]	Budget adherence per ledger entry
run(print_summary=False)	ContextEvalResult	Runs both and aggregates

ContextEvalResult properties: handoff_score, budget_score, overall_score (average of dimensions present), content_loss_detected, plus to_dict().

---

Common Patterns

Gate a workflow in CI, catch handoff regressions, or combine with the Harness Evaluator.

CI gate

import sys
from praisonaiagents.eval import ContextEvaluator

result = ContextEvaluator(
    trace_events=workflow.trace_events,
    agent_order=["researcher", "writer"],
).run()

if result.overall_score < 8:
    sys.exit(1)

Context + accuracy suite

from praisonaiagents.eval import ContextEvaluator, EvalSuite, AccuracyEvaluator

suite = EvalSuite(evaluators=[
    ContextEvaluator(
        trace_events=workflow.trace_events,
        agent_order=["researcher", "writer"],
        budget_ledger=[
            {"agent_name": "researcher", "used_tokens": 8_200, "budget_tokens": 10_000},
            {"agent_name": "writer",     "used_tokens": 4_100, "budget_tokens": 6_000},
        ],
    ),
    AccuracyEvaluator(agent=writer, input_text="...", expected_output="..."),
])
report = suite.run()

---

Best Practices

Feed the full trace, not just the final message

Handoff scoring needs both llm_request and llm_response events for each agent. Pass the complete workflow.trace_events, not a single summary string.

Match agent_order to your workflow order

Scoring walks adjacent pairs in agent_order. List the agents in the exact sequence they run so each handoff is compared correctly.

A budget of 0 or None is neutral, not a fail

Entries with budget_tokens of 0 score a neutral 5.0. Set a real budget only where you want a gate.

LLM-free and deterministic — safe in CI

No live model call runs, so scores are reproducible and require no API key.

---

Related

Evaluation Suite

Run all four evaluators as one CI gate

Evaluation Loop

Iterative agent → judge → improve loop

Judge

LLM-as-judge for evaluating outputs

Harness Evaluator

Score Interactive Test Harness traces

CHL Engineering

The Context / Harness / Loop rubric this evaluator scores against.