> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Event-Loop Watchdog > Detect and self-recover from a wedged asyncio loop in long-lived gateways `LoopWatchdog` detects a wedged asyncio loop and hands the process back to your supervisor before the gateway becomes a silent zombie. **Running `praisonai gateway start`?** You don't need to write any Python β€” the gateway wires the watchdog for you. Add `gateway.watchdog: { enabled: true }` to `gateway.yaml`, or pass `--watchdog` on the CLI. See [Using it from the gateway](#using-it-from-the-gateway) below. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Event-Loop Watchdog" Thread[🧡 OS thread] -->|probe| Loop[πŸ” asyncio loop] Loop -->|ack| Thread Thread --> Check{πŸ’“ alive?} Check -->|yes| Wait[⏳ next probe] Check -->|no Γ— N| Wedge[🚨 dump + exit 75] Wedge --> Sup[πŸ‘· supervisor restart] end classDef input fill:#6366F1,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef check fill:#F59E0B,stroke:#7C90A0,color:#fff classDef alarm fill:#8B0000,stroke:#7C90A0,color:#fff classDef ok fill:#10B981,stroke:#7C90A0,color:#fff class Thread input class Loop process class Check,Wait check class Wedge alarm class Sup ok ``` **Looking for connection heartbeats?** [Gateway Liveness](/docs/docs/features/gateway-liveness) reaps silent **client connections** with `PING`/`PONG` frames. This page covers the **event loop itself** β€” when the process is up but the loop stops running. The two solve orthogonal problems: | Concern | Gateway Liveness | Loop Watchdog | | -------------- | ----------------------------------------- | ----------------------------------------------- | | What is dead? | An individual **client connection** | The **event loop itself** (process is a zombie) | | Detected how? | `PING` / `PONG` frames on the wire | OS-thread probe via `loop.call_soon_threadsafe` | | Reaction | Close the connection (`LIVENESS_TIMEOUT`) | Dump stacks and `os._exit(75)` for restart | | Config surface | `LivenessConfig` in `gateway.yaml` | Python opt-in via `watchdog.arm(loop)` | | Runs on | Gateway reaper task (async) | Dedicated OS thread (survives loop hang) | ## Using it from the gateway Most operators run `praisonai gateway start` β€” turn the watchdog on with YAML or a CLI flag, no Python required. Add a `watchdog` block under `gateway` and start as usual. ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # gateway.yaml gateway: watchdog: enabled: true liveness_interval: 5 # seconds between loop probes liveness_strikes: 3 # hard-exit after N consecutive misses (wedge_after β‰ˆ 15s) dump_file: /var/log/praisonai/loop-wedge.log # optional; stderr always gets the dump ``` ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} praisonai gateway start --config gateway.yaml ``` The gateway arms the watchdog right before `_server.serve()` and disarms it in `finally` and on `stop()`. Pass `--watchdog` to enable it for a single run β€” the flags override the matching YAML fields. ```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # Enable with defaults (probe every 5s, wedge after ~15s) praisonai gateway start --config gateway.yaml --watchdog # Set a specific wedge budget (splits into 3 strikes β†’ probe interval = budget / 3) praisonai gateway start --config gateway.yaml --watchdog --watchdog-timeout 15 # Works without a YAML file too (no-config WebSocket-only mode) praisonai gateway start --watchdog ``` `restart` replays these flags automatically via the persisted start-flags β€” you do not re-pass them. Embedding your own asyncio runtime? Arm the watchdog on the loop directly. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import asyncio from praisonaiagents import Agent from praisonaiagents.gateway import LoopWatchdog agent = Agent(name="Support", instructions="Reply to users.") async def serve(): # ... your long-lived gateway/serving loop here ... await asyncio.sleep(3600) loop = asyncio.new_event_loop() watchdog = LoopWatchdog() # probe every 5s, wedge after 3 misses (~15s) watchdog.arm(loop) try: loop.run_until_complete(serve()) finally: watchdog.disarm() ``` *** ## Quick Start (custom runtime) Build an agent, then arm the watchdog on the loop that serves it. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} import asyncio from praisonaiagents import Agent from praisonaiagents.gateway import LoopWatchdog agent = Agent(name="Support", instructions="Reply to users.") async def serve(): # ... your long-lived gateway/serving loop here ... await asyncio.sleep(3600) loop = asyncio.new_event_loop() watchdog = LoopWatchdog() # probe every 5s, wedge after 3 misses (~15s) watchdog.arm(loop) try: loop.run_until_complete(serve()) finally: watchdog.disarm() ``` Pass a `LoopWatchdogPolicy` to change how fast a wedge is caught. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy watchdog = LoopWatchdog(LoopWatchdogPolicy( probe_interval_s=2.0, # probe every 2s missed_probes_before_wedged=5, # wedge only after ~10s of silence dump_file="/var/log/praisonai/loop-wedge.log", )) ``` Use `dump_only` to catch a hang without killing the process. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy watchdog = LoopWatchdog(LoopWatchdogPolicy(on_wedge="dump_only")) watchdog.arm(loop) # ... later ... if watchdog.wedged: print("loop hung at least once; check the stack dump on stderr") ``` *** ## How It Works A dedicated OS thread probes the loop, so it keeps working precisely when the loop does not. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Sup as Supervisor participant App as Embedder participant WD as LoopWatchdog
(OS thread) participant Loop as asyncio loop App->>WD: arm(loop) loop every probe_interval_s WD->>Loop: call_soon_threadsafe(_ack) alt loop is healthy Loop-->>WD: _ack() runs Note over WD: missed = 0 else loop is wedged Note over Loop: _ack never runs Note over WD: missed += 1 end end Note over WD: missed >= threshold WD->>WD: faulthandler.dump_traceback(all_threads) WD->>Sup: os._exit(75) Sup-->>App: restart process ``` | Piece | Owner | Role | | ----------------------------------------------- | ------------- | ------------------------------------------------------------------------ | | `LoopWatchdogPolicy` | Core (pure) | Configuration: cadence, threshold, on-wedge action, exit code, dump file | | `LoopWatchdog.arm(loop)` | Embedder call | Spawns daemon OS thread `praisonai-loop-watchdog` | | Watchdog thread | Runs off-loop | Schedules `_ack` via `call_soon_threadsafe` every interval | | `faulthandler.dump_traceback(all_threads=True)` | Stdlib | All-thread stack dump on wedge | | `os._exit(75)` | Stdlib | Bypasses `Py_FinalizeEx` β€” supervisor sees `EX_TEMPFAIL` | | `GATEWAY_RESTART_EXIT_CODE` | Protocol | The `75` constant, shared with the restart-intent codes | The watchdog uses `os._exit`, not `sys.exit`, on a confirmed wedge. Normal interpreter shutdown runs `Py_FinalizeEx`, which would itself hang trying to join the wedged loop thread β€” so `os._exit` hands the process straight back to the supervisor, which restarts it cleanly. ### Health signal When the watchdog is configured, `GET /health` (or `gateway.health()` in-process) includes a `watchdog` block. | Field | Type | Description | | ------------------------ | ------- | --------------------------------------------------------------------------------- | | `watchdog.enabled` | `bool` | Always `true` when the block is present (implied by the block existing at all). | | `watchdog.armed` | `bool` | `true` while `_server.serve()` is running; flips to `false` on graceful `stop()`. | | `watchdog.wedge_after_s` | `float` | `liveness_interval * liveness_strikes` β€” the effective wedge budget. | The block is omitted entirely from `health()` when the watchdog isn't configured β€” an always-on gateway sees no shape change. *** ## Configuration Options `LoopWatchdogPolicy` holds every knob. | Option | Type | Default | Description | | ----------------------------- | ---------------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `probe_interval_s` | `float` | `5.0` | Seconds between liveness probes scheduled onto the loop. Also the cadence at which the watchdog thread wakes. Must be finite and `> 0` (raises `ValueError` on `0`, negative, `NaN`, or `inf`). | | `missed_probes_before_wedged` | `int` | `3` | Number of **consecutive** probes that must fail to round-trip within their budget before the loop is declared wedged. Must be `>= 1`. | | `on_wedge` | `"dump_and_exit"` \| `"dump_only"` | `"dump_and_exit"` | What to do once a wedge is confirmed. `"dump_and_exit"` dumps all-thread stacks then calls `os._exit(exit_code)`; `"dump_only"` dumps stacks but leaves the process running. | | `exit_code` | `int` | `75` (`GATEWAY_RESTART_EXIT_CODE`) | Exit code used when `on_wedge == "dump_and_exit"`. Defaults to EX\_TEMPFAIL so a supervisor restarts the process. | | `dump_file` | `Optional[str]` | `None` | Optional path to *also* write the stack dump to (appended, line-buffered). `None` writes to stderr only. | **Computed property:** | Property | Returns | Description | | --------------- | ------- | ------------------------------------------------------------------------------------------------------------------- | | `wedge_after_s` | `float` | `probe_interval_s * missed_probes_before_wedged` β€” approximate stall before a wedge is declared (defaults: `~15s`). | `LoopWatchdog` exposes a small surface: | Method / property | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `LoopWatchdog(policy=None)` | Construct with an explicit `LoopWatchdogPolicy` or accept the defaults. | | `arm(loop)` | Start watching `loop` from a daemon OS thread named `praisonai-loop-watchdog`. Idempotent and fail-open. | | `disarm()` | Stop watching. Safe to call multiple times / when not armed. Joins the worker (best-effort) with a `probe_interval_s + 1.0s` timeout. | | `armed` | `True` iff the worker thread is alive. | | `wedged` | `True` once a wedge has been confirmed (mainly useful for `dump_only`). | ### YAML config keys The `gateway.watchdog` block is a thin faΓ§ade over `LoopWatchdogPolicy` for gateway operators. | YAML key (under `gateway.watchdog.*`) | Type | Default | Maps to | Description | | ------------------------------------- | --------------------------------- | ------- | ------------------------------------------------ | ---------------------------------------------------------------------------------------------------- | | `enabled` | `bool` \| `"true"`/`"yes"`/`"on"` | `false` | β€” | Master switch. Omit or set to `false` and the gateway does not create a watchdog at all β€” zero cost. | | `liveness_interval` | `float` | `5.0` | `LoopWatchdogPolicy.probe_interval_s` | Seconds between probes scheduled onto the loop. | | `liveness_strikes` | `int` | `3` | `LoopWatchdogPolicy.missed_probes_before_wedged` | Consecutive missed probes before a wedge is declared. | | `dump_file` | `str` \| `null` | `null` | `LoopWatchdogPolicy.dump_file` | Optional path to also write the stack dump to. `null` writes to stderr only. | An invalid value (for example `liveness_interval: 0`) fails closed: the gateway logs a warning and runs without a watchdog. ### CLI flag reference * `--watchdog` β€” enable the watchdog for this run (sets/overrides `gateway.watchdog.enabled: true`). * `--watchdog-timeout ` β€” total wedge budget in seconds. Derives `liveness_interval = budget / liveness_strikes` (keeps the default 3 strikes unless YAML overrides). `--watchdog-timeout 15` β†’ probe every 5s. `--watchdog-timeout 6` β†’ probe every 2s. ### Precedence Highest wins: `--watchdog[-timeout]` CLI flag β†’ `gateway.watchdog.*` YAML block β†’ off. Missing values fall back down the chain β€” an unset CLI flag does **not** clobber `enabled: true` in YAML. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB CLI["βš™οΈ --watchdog / --watchdog-timeout"] -->|set| Win[βœ… Watchdog config] CLI -->|unset| YAML["πŸ“„ gateway.watchdog.*"] YAML -->|enabled: true| Win YAML -->|absent / false| Off[⏹️ Off β€” zero cost] classDef cli fill:#6366F1,stroke:#7C90A0,color:#fff classDef yaml fill:#189AB4,stroke:#7C90A0,color:#fff classDef win fill:#10B981,stroke:#7C90A0,color:#fff classDef off fill:#8B0000,stroke:#7C90A0,color:#fff class CLI cli class YAML yaml class Win win class Off off ``` *** ## Choosing a cadence Match the probe interval to how fast a wedge must be caught. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph TB Q[❓ How fast must a wedge be caught?] --> A{"Latency budget"} A -->|"< 5s (realtime chat)"| Fast["probe_interval_s=1, missed=3
wedge in ~3s"] A -->|"~15s (default)"| Def["probe_interval_s=5, missed=3
wedge in ~15s"] A -->|"~60s (batch)"| Slow["probe_interval_s=10, missed=6
wedge in ~60s"] classDef q fill:#6366F1,stroke:#7C90A0,color:#fff classDef opt fill:#10B981,stroke:#7C90A0,color:#fff class Q,A q class Fast,Def,Slow opt ``` Shorter intervals bound recovery time but cost a scheduling roundtrip every N seconds; longer intervals reduce noise and false alerts under GC pauses. A long stop-the-world GC or a big **synchronous** blocking call inside a coroutine looks like a wedge if it exceeds `wedge_after_s`. If you legitimately do 20-second sync work on the loop, either raise `wedge_after_s` above it or β€” better β€” push the work off the loop with `run_in_executor` / `asyncio.to_thread`. `faulthandler` writes one per-thread traceback on a wedge. Read the **loop thread** first (typically `MainThread` or the asyncio worker) β€” that's where the actual hang is. Other threads may just be blocked waiting on that one. *** ## Fail-open guarantees A misbehaving watchdog must never kill a healthy gateway. * `arm()` while already armed is a no-op. * `disarm()` when not armed is safe. * Any exception inside probe scheduling, dumping, or exiting is swallowed. * A closed / stopped loop is **not** a wedge β€” scheduling onto it is a normal shutdown, so the missed counter resets and the watchdog keeps failing open. * A `disarm()` racing an in-flight wedge suppresses the destructive `os._exit` β€” a deliberate shutdown never triggers a supervisor restart. *** ## Common Patterns Three ways teams wire the watchdog. **Long-lived embedder (production):** keep the defaults and wire the process to your existing supervisor (systemd / Kubernetes / pm2). `EX_TEMPFAIL (75)` is the same exit code the rest of the restart-intent protocol uses, so any supervisor that already restarts the gateway handles this too. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.gateway import LoopWatchdog watchdog = LoopWatchdog() # defaults trip at ~15s watchdog.arm(loop) ``` **Diagnostic-only mode (staging):** catch a hang without killing the process, inspect the dump, then flip to `dump_and_exit` in production. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy watchdog = LoopWatchdog(LoopWatchdogPolicy( on_wedge="dump_only", dump_file="/var/log/praisonai/loop-wedge.log", )) watchdog.arm(loop) ``` **Custom exit code for a bespoke supervisor:** override `exit_code` if you can't use `75`. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents.gateway import LoopWatchdog, LoopWatchdogPolicy watchdog = LoopWatchdog(LoopWatchdogPolicy(exit_code=42)) watchdog.arm(loop) ``` *** ## Best Practices systemd's `Restart=on-failure`, launchd's `KeepAlive`, Docker's `restart: unless-stopped`, and Kubernetes all handle exit code 75 correctly. If your supervisor doesn't, the watchdog will still dump stacks but the process won't come back β€” either fix the supervisor or use `on_wedge="dump_only"` for diagnostics. CLI flags survive `restart` via persisted start-flags, but a colleague reading `gateway.yaml` learns the watchdog is on. The YAML block is the discoverable configuration surface. Short-lived CLI runs don't need the watchdog β€” the process exits anyway. Arm it around gateways, bots, and other loops that must stay up for hours. The watchdog is a daemon thread and never blocks shutdown, but `disarm()` also joins it cleanly and prevents any race with a shutdown-time wedge. A 100ms interval false-positives under GC or big JSON serialisation. Keep `probe_interval_s` above your longest expected synchronous block on the loop. systemd (`Restart=on-failure`) restarts on `75`; a naive `nohup` does not. See [Exit Codes](/docs/docs/features/gateway-exit-codes) for the full restart-intent contract. *** ## Related Connection-level heartbeats that reap silent clients (disambiguation). The shared `EX_TEMPFAIL (75)` restart-intent protocol. What to keep on unhealthy exits. Guard against restart storms. Re-drive interrupted turns after the restart.