> ## Documentation Index > Fetch the complete documentation index at: https://praison.ai/docs/llms.txt > Use this file to discover all available pages before exploring further. # Outbound Resilience > Retry transient send failures on every bot channel and park permanently-failed replies in a per-platform durable DLQ — safe by default, no config needed. Outbound Resilience makes sure your agent's reply actually reaches the user on every channel — retrying transient send errors with bounded backoff, and **parking permanently-failed replies in a durable per-platform DLQ by default** so they can be replayed instead of silently lost. No configuration is required. **Safe by default (as of PraisonAI #3447 / PR #3446).** The outbound reply is a durable delivery obligation by default — symmetric with the inbound journal. A permanent or exhausted send failure is **automatically parked** at `~/.praisonai/state//outbound_dlq.sqlite` so it can be replayed, without any configuration. To disable durable parking entirely, set `outbound_resilience.enabled = false`. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent from praisonai.bots import SlackBot agent = Agent( name="Support Bot", instructions="Answer customer questions politely.", ) bot = SlackBot(token="xoxb-...", agent=agent) bot.start() # If the gateway crashes mid-send, the reply is parked in the outbound DLQ. # On the next boot, it is automatically redelivered with a marker: # # ♻️ Recovered after restart — this reply may be a duplicate. # # # # No operator step, no code change — symmetric with the inbound journal. ``` The user sends a message on Slack; transient send failures retry with backoff until the reply is delivered or parked in the DLQ — and a parked reply is automatically redelivered on the next gateway boot. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} graph LR subgraph "Outbound Resilience" Agent[🤖 Agent] --> Send[📤 Send Reply] Send -->|transient 429/503| Backoff[⏳ Backoff Retry] Backoff --> Send Send -->|success| Success[✅ Delivered] Backoff -->|max attempts| DLQ["💾 DLQ Park\n(~/.praisonai/state/<platform>/)"] DLQ --> Replay[🔁 Replay Later] end classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff classDef process fill:#189AB4,stroke:#7C90A0,color:#fff classDef retry fill:#F59E0B,stroke:#7C90A0,color:#fff classDef success fill:#10B981,stroke:#7C90A0,color:#fff classDef storage fill:#6366F1,stroke:#7C90A0,color:#fff class Agent agent class Send process class Backoff retry class Success success class DLQ,Replay storage ``` ## Quick Start Every bot adapter retries transient failures with backoff **and** parks a permanent / exhausted failure at a canonical per-platform DLQ so nothing is silently lost — with no config on your side. ```python theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} from praisonaiagents import Agent from praisonai.bots import SlackBot agent = Agent( name="Support Bot", instructions="Answer customer questions politely.", ) bot = SlackBot(token="xoxb-...", agent=agent) bot.start() # Transient Slack 5xx / rate-limit responses are retried. # A permanently-failed reply (bad channel, revoked bot, etc.) is parked at # ~/.praisonai/state/slack/outbound_dlq.sqlite # for later replay — with no config on your side. ``` Set `outbound_resilience` in `praisonai.yml` only to override defaults — e.g. more attempts, a longer cap, or a custom DLQ path. ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: slack: outbound_resilience: initial_ms: 1000 max_ms: 30000 factor: 2.0 max_attempts: 5 jitter: 0.25 # dlq_path: /var/lib/praisonai/slack-dlq.sqlite # optional override ``` The default DLQ path is `~/.praisonai/state//outbound_dlq.sqlite` — only set `dlq_path` if you need it elsewhere (e.g. a shared volume in Docker). `enabled: false` disables **both** retry and durable parking on that channel — a permanently-failed reply on that channel is lost, matching the pre-safe-by-default behaviour. ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: email: outbound_resilience: enabled: false ``` *** ## How It Works ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Agent as 🤖 Agent participant Adapter as 📡 Bot Adapter participant Platform as 📱 Platform API participant DLQ as 💾 DLQ Agent->>Adapter: deliver_outbound(reply) Adapter->>Platform: send (attempt 1) Platform-->>Adapter: 429 Too Many Requests (Retry-After: 2s) Note over Adapter: Honour Retry-After, wait 2s Adapter->>Platform: send (attempt 2) Platform-->>Adapter: 503 Service Unavailable Note over Adapter: Exponential backoff Adapter->>Platform: send (attempt 3) Platform-->>Adapter: 200 OK Adapter-->>Agent: ✅ success alt Max attempts exhausted Adapter->>DLQ: park(channel_id, reply_text, thread_id) Adapter-->>Agent: ❌ exception re-raised end ``` When a send fails, the adapter classifies the error: | State | What happened | What happens next | | ----------------------------------- | --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | **Success** | Platform returned 2xx | Done — no further action | | **Transient retry** | 429 / 503 / network error | Wait (honouring `Retry-After`) and retry | | **Exhausted → DLQ** | Max attempts reached on transient error | Parked in the default DLQ (`~/.praisonai/state//outbound_dlq.sqlite`), or the custom `dlq_path`; exception re-raised | | **Permanent error → DLQ** | 4xx (not 429) | Parked immediately; exception re-raised | | **Storage transiently unavailable** | DLQ init failed on this send | Degrades to retry-only for **this** send; state does not latch — the next send re-attempts DLQ init and parks the reply once storage recovers | | **`enabled: false`** | Operator opted out | One attempt, no retry, no parking; original exception re-raised | *** ## Automatic crash-recovery on start Every gateway-managed channel redelivers any reply parked by a crash on a previous run — no operator step, symmetric with the inbound journal's `replay()` sweep. The gateway schedules `WebSocketGateway._replay_outbound_dlq(bot)` about 5 seconds after each channel task starts (a grace period for the transport to connect), then calls `OutboundDLQ.redeliver(send)` — draining parked replies oldest-first through the adapter's own `send_message`, so the platform's normal retry/backoff still applies. A redelivery that fails again is kept for the next boot, bounded by the DLQ's existing TTL / `max_size` / `attempts` invariants. Failure never blocks channel start — the behaviour degrades to today's "parked until manual replay". Users receive redelivered replies with a **visible duplicate marker** so an ambiguous redelivery is never silent: > ♻️ Recovered after restart — this reply may be a duplicate. > > \ The marker is prepended idempotently — a parked reply that already starts with it is not double-marked. When something is recovered you'll see this line in the gateway log: ``` Outbound crash-recovery: redelivered N parked repl(y/ies) on start (marked as possible duplicates) ``` ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Boot as 🛠️ Gateway boot participant Bot as 📡 Adapter participant DLQ as 💾 OutboundDLQ participant Platform as 📱 Platform API participant User as 👤 User Boot->>Bot: start_channels() Note over Boot,Bot: 5s grace for transport connect Boot->>DLQ: _replay_outbound_dlq(bot) DLQ->>DLQ: redeliver(send) — oldest first loop each parked reply DLQ->>Bot: send_message(♻️ marker + text) Bot->>Platform: send alt success Platform-->>Bot: 200 OK Bot-->>User: recovered reply DLQ->>DLQ: delete entry else re-fails Platform-->>Bot: transient error DLQ->>DLQ: keep entry, attempts++ end end ``` Only the exact reply the drainer is currently redelivering is suppressed from re-parking. A genuinely-new reply that fails **concurrently** during the boot drain is still parked — the durable-outbound contract holds even mid-recovery. Automatic crash-recovery introduced in [PraisonAI #3862](https://github.com/MervinPraison/PraisonAI/issues/3862). Separate from this boot-time DLQ drainer, `ChannelSupervisor` also re-drains the durable outbox when a channel recovers from a transient outage — see [Channel supervision → Recovery-triggered outbox re-drain](/docs/features/gateway-channel-supervision#recovery-triggered-outbox-re-drain). *** ## Which Channels Does This Apply To? Both retry/backoff **and** durable DLQ parking are on by default for every adapter — no `dlq_path` opt-in required. | Channel | Retry/backoff | DLQ on exhaust | | --------- | ------------- | -------------- | | Telegram | ✅ default-on | ✅ default-on¹ | | Slack | ✅ default-on | ✅ default-on¹ | | Discord | ✅ default-on | ✅ default-on¹ | | WhatsApp | ✅ default-on | ✅ default-on¹ | | Email | ✅ default-on | ✅ default-on¹ | | Linear | ✅ default-on | ✅ default-on¹ | | AgentMail | ✅ default-on | ✅ default-on¹ | ¹ Default DLQ location: `~/.praisonai/state//outbound_dlq.sqlite`. Override with `outbound_resilience.dlq_path`; disable with `outbound_resilience.enabled = false`. **AgentMail additionally deduplicates at the provider.** `AgentMailBot` forwards a stable `Idempotency-Key` header on every attempt of a logical send. When retry crosses the AgentMail boundary after a timeout-after-send, the provider dedupes — no duplicate email is delivered. See [Duplicate protection on retries](/docs/features/email-bot#duplicate-protection-on-retries). **Safe by default ([PR #3447](https://github.com/MervinPraison/PraisonAI/pull/3447)).** The outbound DLQ is now enabled on every adapter without any configuration — mirroring the inbound journal, which has been durable-by-default since PraisonAI #1915. A permanent send failure is parked at `~/.praisonai/state//outbound_dlq.sqlite` for replay instead of silently dropped. The operator escape hatch is `outbound_resilience.enabled = false`. Both **text and media** now inherit the same policy: | Send path | Retry/backoff | Honours `Retry-After` | | ------------------------------------------ | ------------------------------- | --------------------- | | Text reply (`send_message`) | ✅ via `OutboundResilienceMixin` | ✅ | | Media upload (`DeliveryRouter.send_media`) | ✅ via `deliver_with_retry` | ✅ | *** ## Media uploads share this policy The same retry wrapper covers outbound **media** uploads (`DeliveryRouter.send_media` → Telegram `send_photo`, Slack `files_upload_v2`, Discord file send). A transient transport failure on an image or file attachment is retried with the same bounded backoff and `Retry-After` honouring as the text path. * The adapter's configured `_outbound_backoff` `BackoffPolicy` is reused; if the adapter has no policy, a default of `initial_ms=1000`, `max_ms=10000`, `factor=1.5`, `max_attempts=3` applies. * Adapters with no upload primitive still return `False` immediately — no wasted retries. * Permanent errors surface as before after the attempt budget is spent. See [Outbound Media Delivery → Retry & backoff on transient failures](/docs/docs/features/outbound-media-delivery#retry--backoff-on-transient-failures) for the media-specific sequence diagram and behaviour matrix. *** ## Configuration Options All settings live under `outbound_resilience` in your channel config (in `praisonai.yml` or via `BotConfig`). | Option | Type | Default | Description | | -------------- | ------------- | --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `enabled` | `bool` | `True` | Set to `False` to opt this channel out entirely — one attempt, no retry, no DLQ. | | `initial_ms` | `int` | `1000` | First backoff delay in milliseconds. | | `max_ms` | `int` | `10000` | Maximum backoff delay (caps exponential growth). | | `factor` | `float` | `1.5` | Multiplicative growth factor per retry. | | `max_attempts` | `int` | `3` | Total attempts before parking (1 = no retry). | | `jitter` | `float` | `0.25` | Random jitter fraction (0.0–1.0) applied to each delay. | | `dlq_path` | `str \| None` | `~/.praisonai/state//outbound_dlq.sqlite` | Where permanent / exhausted failures are parked. Defaults to the canonical per-platform store (mirrors the inbound journal). Set to override for a shared volume, a custom filesystem, or a distinct file per instance. | The canonical default path is resolved via `resolve_durable_store_dir()`. Set the `PRAISONAI_HOME` environment variable to relocate the whole `~/.praisonai/` root (useful in tests or containers). Before PraisonAI #3447 you had to set `dlq_path` to opt into a DLQ park. On #3447+ that step is unnecessary — the default path is already applied. Existing configs that set `dlq_path` continue to work unchanged (an explicit path always wins over the default). Full YAML example: ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: slack: outbound_resilience: # dlq_path is optional — defaults to ~/.praisonai/state/slack/outbound_dlq.sqlite max_attempts: 5 max_ms: 30000 factor: 2.0 discord: outbound_resilience: # Custom DLQ path for a shared volume dlq_path: /var/lib/praisonai/state/discord/outbound_dlq.sqlite max_attempts: 3 ``` *** ## Transient DLQ Init Recovery If the default DLQ path is briefly unwritable when the adapter tries to initialise it (SQLite lock, disk permission race, transient FS error), Outbound Resilience does **not** latch itself into a permanently-degraded state. The failed send this turn is retried at the transport level and the DLQ init is re-attempted on the **next** send — so a first-turn storage blip does not silently disable durable parking for the life of the process. ```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} sequenceDiagram participant Send as 📤 send #1 participant Init as 🔧 DLQ init participant Storage as 💾 Storage participant Send2 as 📤 send #2 Send->>Init: build OutboundDLQ(path) Init->>Storage: open sqlite Storage-->>Init: OSError (transient) Note over Init: log warning
degrade THIS send
do NOT latch _ready Init-->>Send: retry-only path (no park) Send-->>Send: exception re-raised Note over Storage: storage recovers Send2->>Init: build OutboundDLQ(path) (retry) Init->>Storage: open sqlite Storage-->>Init: ok Init-->>Send2: DLQ ready, park permanent failures ``` You will see a `Failed to initialize outbound DLQ (will retry on next send): ` warning in the log the first time this happens; a subsequent send will silently succeed if storage has recovered. *** ## Common Patterns **Opt one channel out entirely:** ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: email: outbound_resilience: enabled: false # one attempt, no retry, no DLQ park ``` **Use a shared DLQ path (e.g. a Docker volume):** ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: slack: outbound_resilience: dlq_path: /var/lib/praisonai/state/slack/outbound_dlq.sqlite ``` **Aggressive retry for a flaky upstream (defaults handle 99% of cases):** ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} bots: whatsapp: outbound_resilience: max_attempts: 5 max_ms: 30000 factor: 2.0 ``` **DLQ replay** — failed replies parked in the SQLite DLQ can be replayed with the same tooling as the inbound DLQ. See [Inbound DLQ](/docs/features/inbound-dlq) for the analogous replay command. *** ## Best Practices The default DLQ path is `~/.praisonai/state//outbound_dlq.sqlite`. If your process runs as a user with no home directory, or `~/.praisonai/` is on a container tmpfs, parked failures are still logged but lost on restart — set `PRAISONAI_HOME` to a persistent path or override `dlq_path` explicitly. The defaults (`initial_ms=1000`, `max_ms=10000`, `max_attempts=3`) handle the vast majority of transient failures. Only adjust them after observing DLQ entries accumulating in your SQLite file — that signals the defaults are too aggressive or too conservative for your traffic. The default `~/.praisonai/state//outbound_dlq.sqlite` is already persistent on any conventional deployment. Set `dlq_path` when you need a shared filesystem (multi-replica) or a distinct file per instance. ```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}} # ✅ Default — durable, per-platform, isolated # (omit dlq_path) # ✅ Shared volume — multi-replica setup dlq_path: /var/lib/praisonai/state/slack/outbound_dlq.sqlite # ❌ Bad — lost on container restart dlq_path: /tmp/slack-dlq.sqlite ``` Slow retries usually mean the platform backoff is long (e.g., `Retry-After: 60`). The mixin already honours `Retry-After` headers automatically. Increase `max_ms` to allow longer waits rather than disabling resilience entirely — `enabled: false` also throws away durable parking, which is almost never what you want. Outbound Resilience now parks permanent and exhausted failures automatically. You don't need to set `dlq_path` or opt in to a separate "durable delivery" mode — the mixin uses the same canonical `~/.praisonai/state//outbound_dlq.sqlite` store as the inbound journal. Set `outbound_resilience.enabled = false` only if you genuinely want the pre-#3447 retry-without-park behaviour (fire-and-forget channels, ephemeral test bots). Once a permanent / exhausted failure is parked in the DLQ, the gateway automatically redelivers it on the next boot (labelled `♻️ Recovered after restart — this reply may be a duplicate.`). See [Automatic crash-recovery on start](#automatic-crash-recovery-on-start). The only remaining loss window is a crash that hits **after** the send failed but **before** the entry was persisted — the same narrow window covered by Durable Delivery's SQLite outbox. Reach for [Durable Delivery](/docs/features/durable-delivery) when you need that pre-park guarantee; for the typical exhausted-retry case, Outbound Resilience alone is now crash-safe. *** ## Related SQLite outbox for crash-safe delivery with send\_durable() and startup drain Dead-letter queue for failed inbound message processing Per-user and per-channel rate limiting for bot commands Full reference for all per-channel configuration options