Dynamic Judge

Evaluate agent outputs against custom criteria for any domain

Score agent responses against your own criteria — recipe quality, data pipelines, manufacturing checks, or any custom rubric.

from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(name="assistant", instructions="Answer briefly and accurately")
output = agent.start("What is the capital of France?")

result = Judge(criteria="Answer is correct and concise").run(output=output)
print(f"Score: {result.score}/10")

The user runs the agent; the judge scores the final output against your custom criteria and returns a score with reasoning.

How It Works

Quick Start

1. Simple criteria

   from praisonaiagents import Agent
   from praisonaiagents.eval import Judge
   
   agent = Agent(name="assistant", instructions="Be helpful")
   output = agent.start("Explain photosynthesis in one sentence")
   
   judge = Judge(criteria="Response is accurate and concise")
   result = judge.run(output=output)
   print(result.score, result.reasoning)
   
2. Domain-specific config

   from praisonaiagents.eval import Judge, JudgeCriteriaConfig
   
   config = JudgeCriteriaConfig(
       name="data_pipeline",
       description="Evaluate pipeline output",
       prompt_template="Evaluate this output:\n{output}\n\nScore 1-10 for data integrity and performance.",
       scoring_dimensions=["integrity", "performance"],
       threshold=7.0,
   )
   
   judge = Judge(criteria_config=config)
   result = judge.run(output="ETL complete: 1.2M rows, 0 errors")
   
3. CLI recipe optimisation

   praisonai recipe optimize my-recipe --criteria "No bottlenecks, validated output schema"
   

---

Configuration Options

JudgeCriteriaConfig

Field	Type	Default	Description
name	str	—	Criteria configuration name
description	str	—	What is being evaluated
prompt_template	str	—	Prompt with {output}, {input}, {expected} placeholders
scoring_dimensions	List[str]	—	Dimensions to score
threshold	float	7.0	Passing score

JudgeConfig

Field	Type	Default	Description
model	str	gpt-4o-mini	LLM for judging
temperature	float	0.1	Sampling temperature
max_tokens	int	500	Max response tokens
threshold	float	7.0	Passing score
criteria	str	None	Simple criteria string

---

Custom Optimisation Rules

Register domain-specific fix patterns with add_optimization_rule:

from praisonaiagents.eval import add_optimization_rule

class WaterLeakRule:
    name = "water_leak"
    pattern = r"(leak|overflow|pressure.drop)"
    severity = "critical"

    def get_fix(self, context):
        return f"Check valve at {context.get('location', 'unknown')}"

add_optimization_rule("water_leak", WaterLeakRule)

Function	Description
add_optimization_rule(name, rule_class)	Register a rule
get_optimization_rule(name)	Get rule by name
list_optimization_rules()	List registered rules

---

Best Practices

Define measurable criteria

Vague rubrics like “good output” produce inconsistent scores. Use specific, testable dimensions.

Calibrate thresholds

Use 8–9 for production-critical checks; 6–7 for exploratory workflows.

Test on known outputs

Run the judge on good and bad examples before wiring it into optimisation loops.

Use scoring dimensions

Break evaluation into dimensions for clearer feedback and targeted fixes.

---

Related

CLI Eval

Run evaluations from the command line

Evaluator Optimiser

Optimise agents with evaluator feedback loops