Run Until • AI Agent SDK

run_until: Run agent iteratively until output meets quality criteria.

run_until

Method

This is a method of the Agent class in the agent module.

Run agent iteratively until output meets quality criteria.

This method implements the “Ralph Loop” pattern: run agent → judge output → improve based on feedback → repeat until threshold met.

Signature

def run_until(prompt: str, criteria: str, threshold: float, max_iterations: int, mode: str, on_iteration: Optional[Callable[[Any], None]], verbose: bool) -> 'EvaluationLoopResult'

Parameters

prompt · str · required

The prompt to send to the agent

criteria · str · required

Evaluation criteria for the Judge (e.g., “Response is thorough”)

threshold · float · default: "8.0"

Score threshold for success (default: 8.0, scale 1-10)

max_iterations · int · default: "5"

Maximum iterations before stopping (default: 5)

mode · str · default: "'optimize'"

“optimize” (stop on success) or “review” (run all iterations)

on_iteration · Optional

Optional callback called after each iteration

verbose · bool · default: "False"

Enable verbose logging

Returns

Returns · 'EvaluationLoopResult'

EvaluationLoopResult with iteration history and final score

Usage

agent = Agent(name="analyzer", instructions="Analyze systems")
    result = agent.run_until(
        "Analyze the auth flow",
        criteria="Analysis is thorough and actionable",
        threshold=8.0,
    )
    print(result.final_score)  # 8.5
    print(result.success)      # True

Uses

• EvaluationLoop
• loop.run

Source

View on GitHub

praisonaiagents/agent/agent.py at line 2371