LLM-as-Judge

Evaluate agent outputs using LLM-based quality assessment

from praisonaiagents import Agent

judge = Agent(name="judge", instructions="Evaluate agent outputs for quality and accuracy.")
agent = Agent(name="writer", instructions="Write a short essay.")
output = agent.start("Write about climate change.")
judge.start(f"Rate this response from 1-10: {output}")

Use an LLM to evaluate and score agent outputs for accuracy, quality, and custom criteria.

The user runs a task agent, then the judge agent scores the output against your rubric.

Quick Start

1. Evaluate Output

   Python

   from praisonaiagents.eval import Judge
   
   # Simple accuracy check
   result = Judge().run(output="4", expected="4", input="What is 2+2?")
   print(f"Score: {result.score}/10, Passed: {result.passed}")
   

   TypeScript

   import { Judge } from 'praisonai';
   
   // Simple accuracy check
   const result = await new Judge().run({
     output: "4",
     expected: "4", 
     input: "What is 2+2?"
   });
   console.log(`Score: ${result.score}/10, Passed: ${result.passed}`);
   

   CLI

   # Judge with expected output
   praisonai eval judge --input "The answer is 4" --expected "4"
   
   # Judge with criteria
   praisonai eval judge --input "Hello, how can I help?" --criteria "Response is helpful"
   

How It Works

Phase	What happens
1. Produce	The task agent generates an output
2. Submit	You pass the output plus expected or criteria to Judge.run()
3. Score	The judge LLM rates the output on a 1–10 scale
4. Return	JudgeResult reports the score, pass/fail, and improvement suggestions

Choosing an Evaluation Approach

Match the judge setup to how strict and comparative your scoring needs to be.

Evaluation Modes

Accuracy Evaluation

Compare agent output against an expected result:

Python

from praisonaiagents.eval import Judge

judge = Judge()
result = judge.run(
    output="Python is a high-level programming language",
    expected="Python is a programming language",
    input="What is Python?"
)

print(f"Score: {result.score}/10")
print(f"Reasoning: {result.reasoning}")

TypeScript

import { Judge } from 'praisonai';

const judge = new Judge();
const result = await judge.run({
  output: "Python is a high-level programming language",
  expected: "Python is a programming language",
  input: "What is Python?"
});

console.log(`Score: ${result.score}/10`);
console.log(`Reasoning: ${result.reasoning}`);

Criteria Evaluation

Evaluate output against custom criteria:

Python

from praisonaiagents.eval import Judge

judge = Judge(criteria="Response is helpful, accurate, and concise")
result = judge.run(output="Hello! I'm here to help you with any questions.")

if result.passed:
    print("✅ Output meets criteria")
else:
    print("❌ Output needs improvement")
    for suggestion in result.suggestions:
        print(f"  • {suggestion}")

TypeScript

import { Judge } from 'praisonai';

const judge = new Judge({ 
  criteria: "Response is helpful, accurate, and concise" 
});
const result = await judge.run({
  output: "Hello! I'm here to help you with any questions."
});

if (result.passed) {
  console.log("✅ Output meets criteria");
} else {
  console.log("❌ Output needs improvement");
  result.suggestions.forEach(s => console.log(`  • ${s}`));
}

Recipe/Workflow Evaluation

Evaluate multi-agent workflow execution:

Python

from praisonaiagents.eval import RecipeJudge

judge = RecipeJudge(mode="context")  # or "memory", "knowledge"
result = judge.run(
    output="Final workflow output...",
    expected="Complete analysis with citations"
)

TypeScript

import { RecipeJudge } from 'praisonai';

const judge = new RecipeJudge({ mode: "context" });
const result = await judge.run({
  output: "Final workflow output...",
  expected: "Complete analysis with citations"
});

Configuration

JudgeConfig

model · string · default: "gpt-4o-mini"

LLM model to use for evaluation

temperature · number · default: "0.1"

Temperature for consistent scoring (lower = more consistent)

maxTokens · number · default: "500"

Maximum tokens for LLM response

threshold · number · default: "7.0"

Score threshold for passing (1-10 scale)

criteria · string

Custom evaluation criteria

Python

from praisonaiagents.eval import Judge, JudgeConfig

config = JudgeConfig(
    model="gpt-4o",
    temperature=0.0,
    threshold=8.0,
    criteria="Response must be technically accurate"
)

judge = Judge(config=config)

TypeScript

import { Judge, JudgeConfig } from 'praisonai';

const config: JudgeConfig = {
  model: "gpt-4o",
  temperature: 0.0,
  threshold: 8.0,
  criteria: "Response must be technically accurate"
};

const judge = new Judge({ config });

JudgeResult

The result object contains:

Field	Type	Description
score	number	Quality score (1-10)
passed	boolean	True only when a numeric score was parsed AND score >= threshold. An unparseable verdict is always passed=False.
reasoning	string	Explanation for the score
output	string	The judged output
expected	string?	Expected output (if provided)
criteria	string?	Criteria used (if provided)
suggestions	string[]	Improvement suggestions
timestamp	number	When evaluation occurred

Fail-Closed Parsing

If the judge’s LLM response can’t be parsed for a numeric SCORE:, the verdict fails closed.

An unparseable response returns JudgeResult(score=1.0, passed=False, reasoning="Unable to parse response") — or the model’s prose if any was captured. This is deliberate: the previous behaviour returned a fabricated 5.0 that silently passed any threshold ≤ 5. 1.0 fails every threshold, so an unreadable response never masquerades as a pass. A logger.warning records the first 200 characters of the raw text.

Empty responses, JSON {}, and prose-only replies like "the answer is wrong" all fail closed.

Markdown-bolded labels like **SCORE:** 1 (the default output style of most models) and bullet-prefixed labels are tolerated — the parser strips **, *, `, and leading -/> before reading the value. Prompts no longer need to warn the model against bolding.

The safety grader (BaseLLMGrader) already followed this pattern — Judge now matches it. See Safety Evaluation.

Judge with Agent

Evaluate an agent’s response directly:

Python

from praisonaiagents import Agent
from praisonaiagents.eval import Judge

agent = Agent(
    name="Math Helper",
    instructions="You solve math problems"
)

judge = Judge()
result = judge.run(
    agent=agent,
    input="What is 15 * 7?",
    expected="105"
)

print(f"Agent scored: {result.score}/10")

TypeScript

import { Agent, Judge } from 'praisonai';

const agent = new Agent({
  name: "Math Helper",
  instructions: "You solve math problems"
});

const judge = new Judge();
const result = await judge.run({
  agent,
  input: "What is 15 * 7?",
  expected: "105"
});

console.log(`Agent scored: ${result.score}/10`);

Custom Judges

Register Custom Judge

Python

from praisonaiagents.eval import Judge, add_judge, get_judge, list_judges

class CodeQualityJudge(Judge):
    """Judge for evaluating code quality."""
    
    def __init__(self, **kwargs):
        super().__init__(
            criteria="Code is clean, efficient, and well-documented",
            **kwargs
        )

# Register
add_judge("code_quality", CodeQualityJudge)

# Use
JudgeClass = get_judge("code_quality")
judge = JudgeClass()

# List all judges
print(list_judges())  # ['accuracy', 'criteria', 'recipe', 'code_quality']

TypeScript

import { Judge, addJudge, getJudge, listJudges } from 'praisonai';

class CodeQualityJudge extends Judge {
  constructor(options = {}) {
    super({
      ...options,
      criteria: "Code is clean, efficient, and well-documented"
    });
  }
}

// Register
addJudge("code_quality", CodeQualityJudge);

// Use
const JudgeClass = getJudge("code_quality");
const judge = new JudgeClass();

// List all judges
console.log(listJudges()); // ['accuracy', 'criteria', 'recipe', 'code_quality']

From the shell: praisonai eval list (short form) or praisonai eval list-judges — both print the same registered types the Python list_judges() returns.

Domain-Agnostic Evaluation

Use JudgeCriteriaConfig for any domain:

Python

from praisonaiagents.eval import Judge, JudgeCriteriaConfig

# Water flow optimization
config = JudgeCriteriaConfig(
    name="water_flow",
    description="Evaluate water flow optimization",
    prompt_template="""Evaluate the water flow configuration:
{output}

Score based on:
- Flow rate efficiency
- Pressure optimization  
- Resource conservation

SCORE: [1-10]
REASONING: [explanation]
SUGGESTIONS: [improvements]""",
    scoring_dimensions=["flow_rate", "pressure", "efficiency"],
    threshold=7.0
)

judge = Judge(criteria_config=config)
result = judge.run(output="Flow rate: 50L/min, Pressure: 2.5 bar")

TypeScript

import { Judge, JudgeCriteriaConfig } from 'praisonai';

const config: JudgeCriteriaConfig = {
  name: "water_flow",
  description: "Evaluate water flow optimization",
  promptTemplate: `Evaluate the water flow configuration:
{output}

Score based on:
- Flow rate efficiency
- Pressure optimization
- Resource conservation

SCORE: [1-10]
REASONING: [explanation]
SUGGESTIONS: [improvements]`,
  scoringDimensions: ["flow_rate", "pressure", "efficiency"],
  threshold: 7.0
};

const judge = new Judge({ criteriaConfig: config });
const result = await judge.run({ 
  output: "Flow rate: 50L/min, Pressure: 2.5 bar" 
});

Async Evaluation

Python

import asyncio
from praisonaiagents.eval import Judge

async def evaluate_outputs():
    judge = Judge(criteria="Response is helpful")
    
    outputs = [
        "Hello! How can I help?",
        "I don't know.",
        "Let me help you with that!"
    ]
    
    results = await asyncio.gather(*[
        judge.run_async(output=output)
        for output in outputs
    ])
    
    for output, result in zip(outputs, results):
        print(f"{output[:30]}... → {result.score}/10")

asyncio.run(evaluate_outputs())

TypeScript

import { Judge } from 'praisonai';

async function evaluateOutputs() {
  const judge = new Judge({ criteria: "Response is helpful" });
  
  const outputs = [
    "Hello! How can I help?",
    "I don't know.",
    "Let me help you with that!"
  ];
  
  const results = await Promise.all(
    outputs.map(output => judge.runAsync({ output }))
  );
  
  outputs.forEach((output, i) => {
    console.log(`${output.slice(0, 30)}... → ${results[i].score}/10`);
  });
}

evaluateOutputs();

CLI Reference

# Basic judge
praisonai eval judge --input "Output to evaluate"

# With expected output (accuracy mode)
praisonai eval judge --input "4" --expected "4"

# With criteria
praisonai eval judge --input "Hello!" --criteria "Response is friendly"

# Custom threshold
praisonai eval judge --input "Test" --threshold 8.0

# JSON output
praisonai eval judge --input "Test" --json

# Custom model
praisonai eval judge --input "Test" --model gpt-4o

Best Practices

Use Low Temperature

Set temperature: 0.1 or lower for consistent scoring across evaluations.

Define Clear Criteria

Be specific about what constitutes a good output. Vague criteria lead to inconsistent scores.

Set Appropriate Thresholds

• 7.0: Standard quality bar
• 8.0: High quality requirement
• 6.0: Lenient evaluation

Review Suggestions

The suggestions array provides actionable improvements. Use them to iterate on agent prompts.

Related

Evaluation Framework

Complete evaluation suite

Agent Testing

Test your agents