Gateway Crash Forensics

Capture why a gateway died — OOM, supervisor SIGKILL, or parent death — without blocking shutdown

The gateway now ships in the praisonai-bot package. praisonai serve gateway still works exactly as documented here; for a standalone install see praisonai-bot Migration.

When the gateway dies unexpectedly, forensics writes one log line and one diagnostic file so the next boot can tell you why.

from praisonaiagents import Agent
from praisonai.bots import BotOS

agent = Agent(name="Support", instructions="Help users on chat channels.")

BotOS(agent=agent, platforms=["telegram"]).start()

The user keeps chatting until SIGTERM or OOM; forensics captures a snapshot line and a detached diagnostic file for the next operator review.

Quick Start

1. Default behaviour — nothing to configure

   Forensics is on by default. Start the gateway normally:

   praisonai gateway start --config gateway.yaml
   

   The gateway automatically creates ~/.praisonai/gateway/forensics/ at startup and registers signal handlers. No extra steps needed.

2. On the next crash, check the log line

   After a shutdown or crash, search your logs for the gateway-forensics: marker:

   WARNING  gateway-forensics: signal=SIGTERM pid=1234 ppid=1 supervised=yes loadavg_1m=0.50 traced=no maxrss_kb=99000
   

   This single line tells you the signal, whether the process was supervised, system load at crash time, and peak memory use.

3. Read the detached diagnostic file

   The gateway also writes a full diagnostic report that survives a SIGKILL of the process group (started with start_new_session=True):

   ls ~/.praisonai/gateway/forensics/
   # gateway-forensics-1234.log
   
   cat ~/.praisonai/gateway/forensics/gateway-forensics-1234.log
   # # gateway forensics ctx: {...}
   # ## process tree
   # ...
   # ## load
   # 0.50 0.40 0.30 2/123 4567
   # ## recent kernel OOM/killed
   # [Mon Jun 30 ...] Killed process 1234 (python) ...
   

---

How It Works

The gateway sets should_exit = True first, so draining always begins even if the forensic snapshot takes time under memory pressure. The snapshot and diagnostic are fire-and-forget — they never gate the drain.

What ends up in the log line

Key	Source	Meaning
signal	passed by handler	SIGTERM or SIGINT
pid	os.getpid()	Gateway PID
ppid	os.getppid()	Parent PID; 1 means reparented to init
supervised	is_supervised(ppid, $INVOCATION_ID)	yes when systemd or container init
loadavg_1m	os.getloadavg()[0]	1-minute load average
traced	/proc/self/status TracerPid	yes if a debugger was attached
maxrss_kb	getrusage(RUSAGE_SELF).ru_maxrss	Max RSS in kB

Any field that can’t be read is silently omitted from the line.

What ends up in the diagnostic file

Section	Source
# gateway forensics ctx:	The snapshot dict
## process tree	ps -o pid,ppid,stat,rss,etime,cmd --ppid 1
## load	/proc/loadavg
## recent kernel OOM/killed	`dmesg	grep -iE ‘killed process	out of memory	oom’	tail -n 20`

---

Configuration Options

# gateway.yaml
gateway:
  forensics:
    enabled: true                                       # default true
    diagnostic_dir: ~/.praisonai/gateway/forensics      # default
    stop_timeout: 90                                    # optional; only used for the startup warning

Key	Type	Default	Description
gateway.forensics.enabled	bool (or "true"/"false" string)	true	Turn shutdown forensics on/off. String values from env-substituted YAML are honoured.
gateway.forensics.diagnostic_dir	str	~/.praisonai/gateway/forensics	Where detached gateway-forensics-<pid>.log files are written. The directory is created at startup (bounded 2s), not on the signal path.
gateway.forensics.stop_timeout	float (seconds)	None	Your supervisor’s configured stop-timeout. Used only by the startup warning to detect mid-drain SIGKILL risk.

Environment variable:

Var	Equivalent	Default
PRAISONAI_STOP_TIMEOUT	gateway.forensics.stop_timeout	unset

The startup warning

If stop_timeout (from config or PRAISONAI_STOP_TIMEOUT) is less than drain_timeout + 30s, the gateway logs once at startup:

WARNING  Supervisor stop-timeout (40s) < drain_timeout (30s) + headroom; gateway may be killed mid-drain with no explanation.

To silence it: raise your supervisor’s stop-timeout to at least drain_timeout + 30s, or set gateway.forensics.stop_timeout to match your real supervisor config.

Disabling

gateway:
  forensics:
    enabled: false

Set enabled: "false" when substituting from an environment variable. The log-line snapshot is suppressed too — there is no separate switch.

---

Common Patterns

systemd unit

# /etc/systemd/system/praisonai-gateway.service
[Service]
TimeoutStopSec=120
Environment=PRAISONAI_STOP_TIMEOUT=120
ExecStart=praisonai gateway start --config /etc/praisonai/gateway.yaml

Set TimeoutStopSec to at least drain_timeout + 30s. Export PRAISONAI_STOP_TIMEOUT so the startup warning matches reality.

Kubernetes

spec:
  terminationGracePeriodSeconds: 120
  containers:
    - name: gateway
      env:
        - name: PRAISONAI_STOP_TIMEOUT
          value: "120"
      volumeMounts:
        - name: forensics
          mountPath: /home/app/.praisonai/gateway/forensics
  volumes:
    - name: forensics
      persistentVolumeClaim:
        claimName: gateway-forensics-pvc

Mount a persistent volume at diagnostic_dir so reports survive pod termination.

Docker (foreground)

docker run \
  -v /host/forensics:/root/.praisonai/gateway/forensics \
  praisonai gateway start --config /config/gateway.yaml

Bind-mount diagnostic_dir onto the host for inspection after container exit.

---

Programmatic Use

Operators wiring custom supervision can use the pure helpers and the protocol directly:

from praisonai.gateway import (
    ShutdownForensicsProtocol,
    format_forensics_for_log,
    is_supervised,
    drain_timeout_has_headroom,
)

assert is_supervised(ppid=1, invocation_id=None) is True
assert drain_timeout_has_headroom(stop_timeout_s=90, drain_timeout_s=30) is True

class MyForensics:
    def snapshot(self, signal_name=None): ...
    def spawn_diagnostic(self, ctx, log_dir): ...

assert isinstance(MyForensics(), ShutdownForensicsProtocol)

format_forensics_for_log(ctx) renders a snapshot dict as a single log line prefixed with gateway-forensics:. Grep this marker to wire restart-cause dashboards.

---

Best Practices

Leave forensics on

The snapshot is <10ms and never blocks asyncio teardown. The cost of being wrong about a future crash is much higher than the cost of one log line per shutdown.

Match stop_timeout to your supervisor

The startup warning is the only way to learn at boot that your supervisor would SIGKILL you mid-drain. Set gateway.forensics.stop_timeout or PRAISONAI_STOP_TIMEOUT to match your real supervisor config so the check fires when it matters.

Persist diagnostic_dir

A tmpfs or container-local dir loses the report when the container dies. Mount a host path or PVC so diagnostic files are available after restart.

Grep gateway-forensics: in your log scraper

The log marker is stable, single-line, and key=value. Wire it into your log aggregator for restart-cause dashboards:

grep "gateway-forensics:" /var/log/praisonai/gateway.log

Don't rely only on the diagnostic file for fast crashes

The detached subprocess writes asynchronously. On an instant SIGKILL of the whole group there may not be enough time. The single log line is your reliable signal — the diagnostic file provides additional detail when time permits.

---

Related

Gateway Overview

Architecture and Quick Start for the gateway server

Drain Trigger

Control graceful shutdown drain behaviour

Scale to Zero

Suspend the gateway when idle — pay only for active time

Error Handling

Unicode-safe error handling for gateway bot replies

Tracing Hook

Carry the same correlation id onto per-stage spans in your tracer

Event-Loop Watchdog

Dump all-thread stacks and restart when the asyncio loop wedges