> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Durable Tool Runs > Survive a crash mid-tool-loop without re-running side-effecting tools or re-billing the LLM Turn on durability with one flag: the agent journals every model and tool boundary, so a crash mid-loop resumes from the exact step it left off — recorded tool results replay instead of re-executing. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Durable Tool Run" Start[🚀 agent.start] --> T1[🔧 Tool 1 done] T1 --> Crash[💥 Crash] Crash --> Resume[🔁 resume_run_id] Resume --> Replay[📒 Tool 1 from journal] Replay --> Inflight{🔎 Tool 2 in-flight?} Inflight -->|restart_safe=True| T2[🔧 Tool 2 live] Inflight -->|effectful / undeclared| Gate[🛑 NotSafelyResumable] T2 --> Done[✅ Success] end classDef start fill:#8B0000,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef warn fill:#F59E0B,stroke:#7C90A0,color:#fff classDef config fill:#6366F1,stroke:#7C90A0,color:#fff classDef success fill:#10B981,stroke:#7C90A0,color:#fff class Start start class T1,T2,Replay,Resume process class Inflight config class Crash,Gate warn class Done success ``` ## Quick Start Set `durable=True` on `ExecutionConfig`, then capture `agent.last_durable_run_id` after the run and persist it alongside your business record. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent, ExecutionConfig def charge_card(amount: float, customer_id: str) -> dict: return payment_gateway.charge(amount, customer_id) agent = Agent( name="Order Processor", instructions="Charge the card and ship the order.", tools=[charge_card, ship_order, notify_customer], execution=ExecutionConfig( durable=True, journal_path="./journal.db", # optional; defaults to ~/.praisonai/runs/journal.db ), ) result = agent.start("Process order #A-1042 for customer C-99") # Persist alongside your business record so a restart can find it run_id = agent.last_durable_run_id ``` Set `resume_run_id` to the saved id and call `start()` with the **same prompt**. Recorded tool results replay from the journal — side-effecting tools do not fire again. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # SAME agent, SAME prompt agent.execution.resume_run_id = run_id result = agent.start("Process order #A-1042 for customer C-99") # charge_card does NOT fire again — the recorded result replays from the journal. # resume_run_id auto-clears to None once the turn ends terminally. ``` The prompt must match on resume. A different prompt raises `ValueError("Resume prompt does not match ...")`, an unknown id raises `ValueError("Cannot resume unknown durable run ...")`, and a terminal run raises `ValueError("Cannot resume terminal durable run ... (status=...)")`. Gateway agents journal their turns by **default** whenever the session store is durable (the shipped default) — no flag needed. Set the flag explicitly only when you want to pin the choice in `gateway.yaml`: ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} gateway: durable_runs: true # already the default — this just makes it explicit ``` See [Gateway Durable Runs](/docs/features/gateway-durable-runs) for the full operator surface, including the `durable_runs: false` and `reliability: "off"` opt-outs. *** ## How It Works The agent records one event per boundary. On resume the loop is re-driven from the top; journalled steps return their recorded payload instantly, and only un-journalled steps do real work. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant Tool participant Journal User->>Agent: agent.start(prompt) Agent->>Journal: begin run (auto) Agent->>Tool: charge_card() Tool-->>Agent: receipt Agent->>Journal: record result (auto) Note over Agent,Journal: 💥 crash before finishing User->>Agent: set resume_run_id, start(prompt) Journal-->>Agent: replay recorded receipt Agent->>Tool: ship_order() (live — first time) Tool-->>Agent: shipment id Agent->>Journal: finalize succeeded (auto) ``` *** ## Run Outcomes Each terminal outcome maps to an auto-finalize state and controls whether `resume_run_id` clears itself. | Outcome | Auto-finalize | `resume_run_id` auto-clears | | --------------------------------------------- | -------------------------- | --------------------------- | | Success | `succeeded` | Yes | | `ToolExecutionError(is_retryable=False)` | `failed` | Yes | | `ToolExecutionError(is_retryable=True)` | *(none — stays `running`)* | No | | `InterruptedError` / `asyncio.CancelledError` | `cancelled` | Yes | | Streaming `GeneratorExit` | `cancelled` | Yes | A retryable failure leaves the run resumable. A non-retryable failure marks it `failed` and non-resumable. A per-tool `NotSafelyResumable` outcome does not by itself change the run status; the loop continues and the run finalizes normally on the next iteration. *** ## Configuration Options Three fields on `ExecutionConfig` control durability. `Agent.last_durable_run_id` is the read-side companion. | Option | Type | Default | Description | | --------------- | --------------- | --------------------------------------- | ---------------------------------------------------------------------------------------------- | | `durable` | `bool` | `False` | Enable journal writes and resume. Zero overhead when `False` — the journal is not constructed. | | `journal_path` | `Optional[str]` | `None` → `~/.praisonai/runs/journal.db` | SQLite path for the run journal. Use `":memory:"` in tests, a mounted volume in containers. | | `resume_run_id` | `Optional[str]` | `None` | Set to a prior run's id to resume it. Auto-cleared to `None` after any terminal turn. | *** ## Idempotency Contract for Tools Any tool whose signature accepts `idempotency_key` (or `**kwargs`) automatically receives a stable per-call key on every invocation. Non-declaring tools work unchanged. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} def charge_card(amount: float, customer_id: str, idempotency_key: str = "") -> dict: return payment_gateway.charge(amount, customer_id, idem_key=idempotency_key) ``` The key shape is `"{run_id}:{seq}:{function_name}"` — the same on resume, so a tool that dedups on it stays safe even before journal replay records its result. `restart_safe` and `idempotency_key` are complementary: `restart_safe=True` says the tool may be replayed at all; the auto-injected `idempotency_key` gives it a stable per-call key to dedupe on when it is. See [Restart-Safety Contract for Tools](#restart-safety-contract-for-tools) below. *** ## Restart-Safety Contract for Tools A tool can declare whether it is safe to re-run after a crash — on resume, an in-flight tool is only re-executed when this declaration says so, otherwise it surfaces a typed outcome for the operator to reconcile. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB Inflight[🔎 In-flight call on resume] --> Check{restart_safe?} Check -->|True| Replay[🔧 Re-executed live] Check -->|False| Gate[🛑 NotSafelyResumable] Check -->|None / undeclared| Heuristic{Read-only name?} Heuristic -->|Yes| Replay Heuristic -->|Uncertain| Gate classDef start fill:#8B0000,stroke:#7C90A0,color:#fff classDef config fill:#6366F1,stroke:#7C90A0,color:#fff classDef process fill:#10B981,stroke:#7C90A0,color:#fff classDef warn fill:#F59E0B,stroke:#7C90A0,color:#fff class Inflight start class Check,Heuristic config class Replay process class Gate warn ``` Declare it on the decorator: `True` for read-only or idempotent tools, `False` for anything that sends, writes, charges, or deletes. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent, ExecutionConfig, tool @tool(restart_safe=True) def fetch_report(report_id: str) -> dict: return report_api.get(report_id) @tool(restart_safe=False) def send_invoice(customer_id: str, amount: int) -> str: return billing.send(customer_id, amount) agent = Agent( name="Billing Agent", instructions="Fetch the report, then send the invoice.", tools=[fetch_report, send_invoice], execution=ExecutionConfig(durable=True), ) agent.start("Bill customer C-99 for report R-42") ``` | Option | Type | Default | Description | | -------------- | ---------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `restart_safe` | `Optional[bool]` | `None` | Declare replay behaviour for durable resume. `True` = safe to re-run (read-only / idempotent). `False` = effectful; must never be silently re-executed on resume. `None` = undeclared — falls back to the read-only name heuristic and fails closed when uncertain. | Pick the value by asking whether the tool has an external side effect that is not safe to repeat. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB Q1{External side effect?
send / write / charge / delete} -->|No| Safe[restart_safe=True] Q1 -->|Yes| Q2{Idempotent?
dedups on idempotency_key} Q2 -->|Yes| Safe Q2 -->|No| Unsafe[restart_safe=False] classDef config fill:#6366F1,stroke:#7C90A0,color:#fff classDef process fill:#10B981,stroke:#7C90A0,color:#fff classDef warn fill:#F59E0B,stroke:#7C90A0,color:#fff class Q1,Q2 config class Safe process class Unsafe warn ``` The class attribute form `BaseTool.restart_safe` is the equivalent for subclass-style tools. The same declaration also gates the ordinary tool-retry loops: a `restart_safe=False` tool is executed exactly once even when a transient error would normally be retried. See [Tool Retry & Backoff → Retrying Non-Idempotent Tools](/docs/features/tool-retry-backoff#retrying-non-idempotent-tools). *** ## When resume returns `NotSafelyResumable` An effectful or undeclared tool that was in-flight at the crash is not re-run — the resumed turn records a typed outcome instead, so the operator can reconcile the external action. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} { "error": ( "Tool '' was in-flight when the durable run crashed and is not " "declared restart_safe; its outcome could not be confirmed. It was not " "re-executed on resume to avoid a duplicate side effect. Reconcile the " "external action manually or mark the tool @tool(restart_safe=True) if " "it is idempotent." ), "error_type": "NotSafelyResumable", "not_safely_resumable": True, "restart_safe": False, } ``` Reconcile the external action out of band (or with an idempotency check) and re-drive the loop, or mark the tool `restart_safe=True` if it is genuinely idempotent. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant User participant Agent participant Tool participant Journal User->>Agent: agent.start(prompt) Agent->>Tool: send_invoice() Note over Agent,Journal: 💥 crash before result journalled User->>Agent: set resume_run_id, start(same prompt) Journal-->>Agent: send_invoice was in-flight, not restart_safe Agent-->>User: {"error_type": "NotSafelyResumable", ...} — not re-executed ``` *** ## Discovering Resumable Runs When your process restarts and you don't have the `run_id` handy, ask the journal directly for runs still `running`. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.runtime import RunJournal journal = RunJournal("./journal.db") for run_id in journal.interrupted_runs(): print(f"Resumable: {run_id}") # Set agent.execution.resume_run_id = run_id and re-run the original prompt ``` *** ## Advanced Framework and tool authors integrating a custom executor can reach the durable context directly through `praisonaiagents.agent.durable`. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.agent.durable import get_durable_idempotency_key def charge_card(amount: float, customer_id: str) -> dict: key = get_durable_idempotency_key() # stable per-call key, or None outside a durable run return payment_gateway.charge(amount, customer_id, idem_key=key) ``` The module also exposes `begin_durable_run`, `abegin_durable_run`, `end_durable_run`, `get_durable_run`, and `DurableRunContext` for wiring the journal into a bespoke tool loop. *** ## Common Patterns Long-running scheduled job that survives a container restart: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent, ExecutionConfig from praisonaiagents.runtime import RunJournal agent = Agent( name="Nightly Reconciler", instructions="Reconcile pending orders.", tools=[reconcile_order], execution=ExecutionConfig(durable=True, journal_path="/data/praisonai/journal.db"), ) # On startup, resume anything left running by a previous crash for run_id in RunJournal("/data/praisonai/journal.db").interrupted_runs(): agent.execution.resume_run_id = run_id agent.start("Reconcile pending orders.") ``` Batch of paid API calls that must never double-charge — the recorded receipt replays instead of re-calling: ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} agent = Agent( name="Billing Runner", instructions="Charge each pending invoice once.", tools=[charge_card], execution=ExecutionConfig(durable=True), ) result = agent.start("Charge invoice #A-1042") invoice_run_id = agent.last_durable_run_id # store next to the invoice row # After a crash: resume without re-charging agent.execution.resume_run_id = invoice_run_id agent.start("Charge invoice #A-1042") ``` Pause-for-approval flows survive a restart because the approval decision is journalled — see [Durable Approvals](/docs/features/durable-approvals). *** ## Best Practices Capture `agent.last_durable_run_id` right after `start()`/`chat()` and store it next to the order id or ticket id. That's the value you set into `resume_run_id` later. The default `~/.praisonai/runs/journal.db` is lost when a container is recreated. Pass an explicit path on a mounted volume so the journal survives restarts. Opt into the auto-injected `idempotency_key` so a tool that crashes *before* recording its result still dedups on retry. `True` for read-only / idempotent tools; `False` for anything that sends, writes, charges, or deletes. Undeclared tools fall back to a name heuristic and fail closed on uncertainty — safe, but noisier than an explicit declaration. If a resumed turn returns `error_type: "NotSafelyResumable"`, the in-flight side effect was not re-driven. Reconcile the external action, then either mark the tool `restart_safe=True` (if truly idempotent) or continue the turn out of band. `ToolExecutionError(is_retryable=True)` leaves the run resumable; `is_retryable=False` marks it `failed` and non-resumable. Calling `stream.close()` finalizes the run as `cancelled`, so aborting a stream never leaves a ghost `running` row. *** ## Related The SQLite layer under durable runs. Pause-for-approval that survives a restart. Agent execution limits and configuration. Persist gateway sessions across restarts. Enable durable runs for every agent in the WebSocket gateway. Persist async jobs and idempotency keys with SqliteJobStore.