Gateway Troubleshooting

Common gateway issues and step-by-step solutions

Common issues with the PraisonAI Gateway daemon and server, with step-by-step troubleshooting guides.

from praisonaiagents import Agent

agent = Agent(
    name="Gateway Helper",
    instructions="Diagnose PraisonAI Gateway connectivity issues.",
)
agent.start("Gateway health check: is the daemon reachable?")

The user checks gateway status, reads logs, and restarts the daemon until channels respond again.

Common Issues

Port Already In Use / Two Gateways Running

Symptom: Gateway fails to start with a clear error message about port conflicts.

Error: Gateway port 8765 is already in use.

  Another gateway may be running (PID 8864).
  Stop it:  praisonai gateway stop
  Or use a different port:  GATEWAY_PORT=8766 praisonai gateway start

  Only ONE gateway process should poll each Telegram bot token.

1. Check PID lock and port status

   praisonai gateway status
   

   Look for lines like:

   • Gateway PID lock: Process 12345 running (127.0.0.1:8765)
   • Port 127.0.0.1:8765: In use
2. Stop the existing gateway gracefully

   praisonai gateway stop
   

   This sends SIGTERM and waits for graceful shutdown. The gateway drains active sessions for up to 10 seconds — in-flight turns and queued inbox messages are persisted before exit. Sessions that exceed the timeout are force-persisted with a SESSION_END event (had_pending_work, was_executing). Use --force to skip the drain. See Gateway Session Continuity.

3. If gateway is stuck, force stop

   praisonai gateway stop --force
   

   This sends SIGKILL directly (SIGTERM on Windows) for immediate termination.

4. If port is used by non-gateway process

   # Find what's using the port
   lsof -i :8765
   # Linux alternative:
   netstat -ano | findstr :8765
   
   # Either kill that process or use a different port
   GATEWAY_PORT=8766 praisonai gateway start
   # Or:
   praisonai gateway start --port 8766
   
5. Clean up stale lock file (if needed)

   # Check the PID lock file
   cat ~/.praisonai/gateway-127_0_0_1-8765.pid
   
   # Remove stale lock manually (auto-cleaned on next start)
   rm ~/.praisonai/gateway-127_0_0_1-8765.pid
   

   Stale locks are automatically detected and removed, but you can clean them manually if needed.

---

SSL Certificate Verify Failed on Start (Corporate Proxy / MITM)

Symptom: praisonai gateway start --config gateway.yaml fails with an SSLCertVerificationError in gateway doctor, but the same channel token works fine with --no-preflight.

Cause: Your network intercepts TLS with a corporate CA the probe’s HTTP client does not trust yet. The runtime bot adapter is often more permissive, so the token itself is usually fine.

Fix (pick one):

1. Point PraisonAI at your corporate CA (preferred)

   export PRAISONAI_SSL_CA_BUNDLE=/path/to/corp-ca.pem
   praisonai gateway start --config gateway.yaml
   

   PRAISONAI_SSL_CA_BUNDLE overrides any pre-existing SSL_CERT_FILE and REQUESTS_CA_BUNDLE for the probe.

2. Use the standard SSL env vars

   export SSL_CERT_FILE=/path/to/corp-ca.pem   # or REQUESTS_CA_BUNDLE
   praisonai gateway start --config gateway.yaml
   
3. Skip the preflight check

   praisonai gateway start --config gateway.yaml --no-preflight
   

   Skips channel-credential validation on start. Use only when you know the tokens are good.

Preflight soft-fails on SSL-only errors — it prints a warning naming these three env vars and continues to start automatically. You only need the fixes above if you also want a clean gateway doctor run. A mixed SSL + token/network failure still hard-aborts. See Corporate CA bundle (SSL-inspecting networks).

A configured-but-missing CA bundle path prints Warning: CA bundle path '<path>' does not exist — SSL_CERT_FILE / REQUESTS_CA_BUNDLE not updated for probe. and does not touch the SSL env vars. Double-check the path resolves before restarting.

---

Telegram Bot Goes Silent After Restart

Symptom: Gateway /health endpoint returns healthy, but Telegram messages aren’t received after a restart or crash.

Cause: Two processes are polling the same Telegram bot token. Telegram delivers messages to only one poller, causing the bot to appear “silent” when the wrong process gets the messages.

1. Stop all gateway instances

   praisonai gateway stop --force
   

   This ensures all instances are terminated, even if PID files are corrupted.

2. Verify no processes are running

   praisonai gateway status
   

   Should show:

   • Gateway PID lock: No lock file found
   • Port 127.0.0.1:8765: Available
3. Start a single gateway instance

   praisonai gateway start
   

   Only start one instance to ensure exclusive bot token polling.

4. Test bot responsiveness

   Send a message to your Telegram bot. It should respond normally now that only one process is polling the token.

---

Daemon Running But Gateway Unreachable

Symptom: praisonai gateway status shows Daemon service: Running (launchd) but Gateway not reachable at http://127.0.0.1:8765/health.

1. Verify daemon is actually running

   praisonai gateway status --daemon-only
   

   Look for Running status and process ID.

2. Check daemon logs for errors

   praisonai gateway logs
   
   # Or check raw log files:
   # macOS: tail ~/.praisonai/logs/bot-stderr.log
   # Linux: journalctl --user -u praisonai-bot
   

   Look for Python tracebacks or port binding errors.

3. Check port and PID lock status

   praisonai gateway status
   

   This now reports both port usage and PID lock status directly. Look for:

   • Gateway PID lock: Process <pid> running or No lock file found
   • Port 127.0.0.1:8765: In use or Available

   If another process is using the port, use praisonai gateway stop to stop an existing gateway, or choose a different port.

4. Verify PraisonAI version

   praisonai --version
   

   Upgrade to ≥ v4.6.23 if you see older versions - earlier versions had IndentationError bugs fixed in PR #1484.

5. Restart the daemon

   # macOS
   launchctl kickstart -k gui/$(id -u)/ai.praison.bot
   
   # Linux  
   systemctl --user restart praisonai-bot
   
   # Or reinstall completely
   praisonai onboard
   

---

Rapidly Growing Log Files

Symptom: ~/.praisonai/logs/bot-stderr.log grows to multiple MB per minute.

1. Check log file size

   ls -lh ~/.praisonai/logs/bot-stderr.log
   

   If growing rapidly (>1MB/min), this indicates a crash loop.

2. View recent errors

   tail -50 ~/.praisonai/logs/bot-stderr.log
   

   Look for repeated Python tracebacks, especially IndentationError.

3. Stop the daemon

   # macOS
   launchctl unload ~/Library/LaunchAgents/ai.praison.bot.plist
   
   # Linux
   systemctl --user stop praisonai-bot
   
4. Clear logs and restart

   # Clear the log file
   > ~/.praisonai/logs/bot-stderr.log
   
   # Upgrade PraisonAI
   pip install --upgrade praisonai
   
   # Restart daemon
   praisonai gateway install --start
   

---

Daemon Not Installed

Symptom: praisonai gateway status shows Daemon service: Not installed (systemd).

1. Run the onboarding wizard

   praisonai onboard
   

   This installs the daemon service for your platform.

2. Verify installation

   praisonai gateway status --daemon-only
   

   Should show Installed but not running or Running.

3. Start the service

   # Manual start
   praisonai gateway install --start
   
   # Or check platform-specific commands
   praisonai gateway status
   

---

HTTP 500 on Health Endpoint

Symptom: curl http://127.0.0.1:8765/health returns 500 Internal Server Error.

1. Check PraisonAI version

   praisonai --version
   

   Versions before v4.6.23 had AttributeError bugs in the health endpoint.

2. Upgrade PraisonAI

   pip install --upgrade praisonai
   
3. Restart the gateway

   praisonai gateway install --start
   
4. Test health endpoint

   curl http://127.0.0.1:8765/health
   

   Should return JSON with status, uptime, agents, sessions, clients, and channels.

---

Windows: gateway status shows PID lock status: Unavailable

Symptom: praisonai gateway status prints an advisory PID-lock line, followed by a healthy gateway:

PID lock status: Unavailable (kill returned a result with an exception set)
Gateway Status: healthy
  Uptime: ...

On older versions (pre-v4.6.141) the same underlying cause produced a harder failure that hid the status entirely:

ERROR: Error checking gateway server status:  returned a result with an exception set

Root cause: On Windows, os.kill(pid, 0) — used by GatewayPIDLock._is_process_running to poll whether a lock is stale — can raise SystemError (“returned a result with an exception set”) instead of OSError / ProcessLookupError. Before PraisonAI v4.6.141 that exception propagated through GatewayHandler.status and short-circuited the /health probe, so operators saw no gateway status at all.

What v4.6.141 changes:

1. _is_process_running now catches SystemError and ValueError and returns False (process not running).
2. GatewayHandler.status treats PID-lock inspection as advisory only. Any unexpected exception prints PID lock status: Unavailable (<error>) and the command continues to the /health probe.

1. Check your version

   praisonai --version
   

   Upgrade to ≥ v4.6.141 if you still see ERROR: Error checking gateway server status: returned a result with an exception set.

2. Read the line as advisory

   If praisonai gateway status now prints PID lock status: Unavailable (<error>) followed by Gateway Status: healthy, the gateway is running — the PID line is advisory only.

3. Confirm with the health endpoint

   curl http://127.0.0.1:8765/health
   

   A 200 healthy response confirms the gateway is up regardless of the PID-lock line.

4. Inspect the lock file if needed

   # PowerShell
   Get-Content $env:USERPROFILE\.praisonai\gateway-127_0_0_1-8765.pid
   

   The four lines are pid, host, port, timestamp. A stale PID is cleaned up automatically on the next start/stop/status call.

Related: PraisonAI #2844, fixed by PR #2849.

---

Windows: ‘charmap’ codec error from Telegram bot replies

Symptom: Telegram users receive Error: 'charmap' codec can't encode character '⚠' in position N: character maps to <undefined> instead of the real error message.

Root cause: Windows default console encoding is cp1252, not UTF-8. When agent exceptions contained warning symbols (⚠), emoji, or accented text, the error formatter crashed before the real error could be reported.

1. Verify your version contains the fix

   python -c "from praisonai.gateway.unicode_utils import safe_error_message; print('OK')"
   

   If this command runs without error, you have the fix from PR #1754.

2. Upgrade to a fixed version

   pip install --upgrade praisonai
   

   Upgrade to PraisonAI version that contains PR #1754 — Gateway and Telegram bot error handlers now sanitize exception text to ASCII-safe form for transport, while preserving full Unicode in logs.

3. Restart the gateway

   praisonai gateway install --start
   
4. Test the fix

   Users will now see clean error messages instead of charmap crashes:

   • Error: API quota exceeded. Check billing. (was: charmap crash hiding OpenAI 429)
   • Error: Rate limit exceeded. Try again later.
   • Error: Authentication failed. Check API key.
   • Error: Request timeout. Try again.

No workaround needed on supported versions. The previously recommended PYTHONUTF8=1 / PYTHONIOENCODING=utf-8 workaround is no longer required for the bot reply path (still useful for general console output).

---

Permission Denied Errors

Symptom: Daemon fails to start with permission errors in logs.

macOS

# Check LaunchAgent permissions
ls -la ~/Library/LaunchAgents/ai.praison.bot.plist

# Reload LaunchAgent
launchctl unload ~/Library/LaunchAgents/ai.praison.bot.plist
launchctl load ~/Library/LaunchAgents/ai.praison.bot.plist

Linux

# Check systemd user service
systemctl --user status praisonai-bot

# Enable user lingering (allows services to run when not logged in)
sudo loginctl enable-linger $(whoami)

# Restart service
systemctl --user daemon-reload
systemctl --user restart praisonai-bot

Windows

# Run as administrator
# Check Task Scheduler for PraisonAI task
schtasks /Query /TN PraisonAIGateway

# Delete and recreate
praisonai gateway uninstall
praisonai gateway install

---

Clean Reinstall Process

When all else fails, perform a clean reinstall:

1. Stop and uninstall

   praisonai gateway uninstall
   
2. Clear configuration

   # Backup first if needed
   rm -rf ~/.praisonai/logs
   rm -rf ~/.praisonai/config
   
3. Upgrade PraisonAI

   pip install --upgrade praisonai
   
4. Run onboarding

   praisonai onboard
   

   Follow the wizard to reinstall the daemon service.

5. Verify installation

   praisonai gateway status
   

   Should show daemon running and gateway reachable.

---

Client stops reconnecting after agent_not_found

Symptom: GatewayClient logs Connection abandoned: agent_not_found (reconnect paused; not retrying) and stops attempting to connect, even though the gateway is healthy.

Cause: The agent_id supplied in the client’s hello frame is not registered on the gateway. The server emits agent_not_found with next_step: do_not_retry, and the client now correctly stops the reconnect loop instead of backing off forever.

1. Check the agent_id you pass

   GatewayClient(url="ws://localhost:8765", agent_id="assistant")
   #                                                  ^^^^^^^^^ must exist on the server
   
2. List agents registered on the gateway

   curl http://127.0.0.1:8765/health | jq '.agents'
   

   Compare against the id you’re passing.

3. Fix the id or register the agent

   Either update the client’s agent_id, or add the agent to the gateway configuration and restart it. Then call client.connect() again — the fresh call clears any stale backoff floor from the previous cycle.

4. Surface the failure without log scraping

   Setting client.on_reconnect_paused before connect() gives you an immediate callback with (code, next_step) so the failure surfaces without waiting for a log scrape. See Gateway Client → Terminal vs Transient Connect Errors.

---

Reading the hello_error Envelope

When the gateway rejects a connection, it sends a structured hello_error frame before closing. Each (code, next_step) pair maps to a specific operator action.

code	next_step	Action
rate_limited	wait_then_retry	Wait retry_after_seconds then reconnect; if persistent, raise the rate limiter ceiling
origin_not_allowed	do_not_retry	Add the client origin to allowed_origins in gateway.yaml
configuration_error	do_not_retry	External bind without allowed_origins configured; bind to loopback or set the allowlist
auth_required	reauthenticate	Fetch a token (CLI or pairing flow) and reconnect
protocol_unsupported	upgrade_client	Update the client SDK to a newer version
protocol_unsupported	downgrade_client	Server is older than the client; pin the client to an older release or upgrade the server
agent_not_found	do_not_retry	Fix the agent_id in the hello frame

Example envelope received on rate-limit:

{
  "type": "hello_error",
  "code": "rate_limited",
  "message": "Too many connection attempts",
  "next_step": "wait_then_retry",
  "retry_after_seconds": 30,
  "next": "wait_then_retry"
}

See Gateway Handshake Protocol for the full field reference and ConnectRecoveryStep values.

---

Authentication Errors

GatewayStartupError: Cannot bind to 0.0.0.0 without an auth token

Symptom: Gateway fails to start when binding to external interfaces without authentication.

1. Use onboarding wizard

   praisonai onboard
   

   This automatically generates and saves a secure token.

2. Or set token manually

   export GATEWAY_AUTH_TOKEN=$(openssl rand -hex 16)
   praisonai gateway start --host 0.0.0.0
   

UIStartupError: Cannot bind to 0.0.0.0 with default admin/admin credentials

Symptom: Chainlit UI fails to start on external interface with default credentials.

1. Set custom credentials

   export CHAINLIT_USERNAME=myuser
   export CHAINLIT_PASSWORD=mypass
   praisonai chat --host 0.0.0.0
   
2. Or allow defaults for demos (unsafe)

   export PRAISONAI_ALLOW_DEFAULT_CREDS=1
   praisonai chat --host 0.0.0.0
   

My gateway logs show gw_****xxxx instead of the full token

This is intentional for security — tokens are fingerprinted in logs to prevent exposure.

1. Retrieve full token from environment file

   cat ~/.praisonai/.env | grep GATEWAY_AUTH_TOKEN
   
2. Or check environment variables

   echo $GATEWAY_AUTH_TOKEN
   

Magic-link login works in WebSocket but fails over HTTP (or vice versa)

Symptom: One auth method works but the other fails, even with the same token.

Cause (pre-fix): HTTP/magic-link and WebSocket used different secret sources before PR #1744.

1. Upgrade PraisonAI

   Upgrade to PraisonAI ≥ v4.6.47 (ships PR #1744). Config token now exports to env, unifying all auth paths.

2. Restart the gateway

   praisonai gateway restart
   

   Config token precedence is now enforced on restart.

---

Restart After Config Change

When you update bot configuration files, restart the daemon using these OS-specific commands (matching the onboard Done panel):

macOS

launchctl kickstart -k gui/$(id -u)/ai.praison.bot

Linux

systemctl --user restart praisonai-bot

Windows

schtasks /End /TN PraisonAIGateway && schtasks /Run /TN PraisonAIGateway

---

Diagnostic Commands

Quick commands for gathering diagnostic information:

# Full status check
praisonai gateway status

# Daemon-only check (for scripts)
praisonai gateway status --daemon-only

# Recent logs (last 50 lines)
praisonai gateway logs -n 50

# Check the PID lock file
cat ~/.praisonai/gateway-127_0_0_1-8765.pid

# Stop a running gateway
praisonai gateway stop
praisonai gateway stop --force

# Check port usage
lsof -i :8765

# Test health endpoint directly
curl -v http://127.0.0.1:8765/health

# Check PraisonAI version
praisonai --version

# Platform daemon status
# macOS: launchctl list | grep ai.praison
# Linux: systemctl --user status praisonai-bot
# Windows: schtasks /Query /TN PraisonAIGateway

---

Platform-Specific Notes

macOS

LaunchAgent Path: ~/Library/LaunchAgents/ai.praison.bot.plist Log Path: ~/.praisonai/logs/bot-stderr.log

# Manual management
launchctl load ~/Library/LaunchAgents/ai.praison.bot.plist
launchctl unload ~/Library/LaunchAgents/ai.praison.bot.plist
launchctl kickstart -k gui/$(id -u)/ai.praison.bot

# Check if loaded
launchctl list | grep ai.praison

Linux

Service Path: ~/.config/systemd/user/praisonai-bot.service Logs: journalctl --user -u praisonai-bot

# Manual management
systemctl --user start praisonai-bot
systemctl --user stop praisonai-bot
systemctl --user restart praisonai-bot
systemctl --user enable praisonai-bot

# Check status
systemctl --user status praisonai-bot

Windows

Task Path: Task Scheduler → PraisonAIGateway Logs: Windows Event Log

# Manual management via Task Scheduler or:
schtasks /Run /TN PraisonAIGateway
schtasks /End /TN PraisonAIGateway

# Check status
schtasks /Query /TN PraisonAIGateway

---

Config reload did not apply

Symptom: You edited gateway.yaml or sent SIGHUP, but channels or agents still run the old configuration.

Check	What to look for
Log line	A successful reload logs a summary such as reload applied: agents; restart[telegram] — if absent, the watcher may not have fired or SIGHUP was ignored (Windows has no SIGHUP)
Drain window	Channel restarts wait up to gateway.reload_drain_timeout (falls back to drain_timeout) — a slow turn can delay the visible restart
Install	pip install "praisonai[gateway]" enables watchdog; without it, mtime polling (about 5 s) still reloads but more slowly

See Gateway Config Reload for file-watch vs SIGHUP paths and YAML keys.

---

Multi-Channel Troubleshooting

Common failure modes when using multiple bots on the same platform:

Symptom	Cause	Fix
Token ${X} is used by multiple channels	Two channels reference the same env var in bot.yaml	Give each channel its own env var and create a separate bot in @BotFather
Same bot token value is used by multiple channels	Two different env vars resolve to the same actual token	Generate a fresh token from @BotFather for the duplicate channel
Channel '<x>' token '<env>' doesn't follow naming convention	Custom env var name	Rename to PLATFORM_<ROLE>_BOT_TOKEN (e.g. TELEGRAM_CFO_BOT_TOKEN)
Missing token: <env>	Env var referenced in bot.yaml is unset	Add it to ~/.praisonai/.env

Example Fix:

# Problem: Both channels use the same token
# Error: "Token TELEGRAM_BOT_TOKEN is used by multiple channels"

# Solution: Create unique tokens per role
# 1. Create new bot in @BotFather for CFO role
# 2. Add unique environment variable
export TELEGRAM_CFO_BOT_TOKEN="987654321:DEF..."

# 3. Update bot.yaml
channels:
  telegram_cfo:
    platform: telegram
    token: ${TELEGRAM_CFO_BOT_TOKEN}  # Unique token
    routes:
      default: cfo

Validation:

# Check multi-channel configuration
praisonai doctor multi_channel_tokens

# ✅ Expected output:
# "Multi-channel configuration looks good (N channels)"

---

Channel Supervision Issues

Channel goes silent but /health shows running: true

Symptom: Channel stops receiving messages but /health still reports "running": true.

Cause: Pre-PR-2041 behaviour, or the health: block is missing from gateway.yaml — hung sockets are not detected until an exception is raised.

Fix: Add the proactive health block:

gateway:
  health:
    interval: 300
    stale_after: 120
    enabled: true

See Proactive Health Monitoring.

Channel keeps restarting every 5 minutes

Symptom: Channel restarts on a regular interval visible in logs.

Cause: stale_after too low for a quiet channel, or interval too aggressive for a slow remote API.

Fix: Raise stale_after to e.g. 600, or lower max_restarts_per_hour so the cap kicks in and surfaces the issue:

gateway:
  health:
    stale_after: 600
    max_restarts_per_hour: 3

Channel Shows state: failed in /health

Symptom: When checking GET /health, a channel shows "state": "failed" with error details.

Common Causes:

• Telegram Conflict: Multiple bot instances using the same token
• Invalid Credentials: Bot token revoked or incorrect
• Permission Issues: Bot lacks required permissions

Investigation Steps:

1. Check Error Details

   curl http://127.0.0.1:8765/health | jq '.channels.telegram.supervision'
   

   Look for last_error and last_error_time fields.

2. Telegram Conflict Resolution

   If error contains “Conflict: terminated by other getUpdates”:

   # Stop any other bot instances using the same token
   # Then force reconnect to reset state
   praisonai gateway reconnect telegram
   
3. Credential Verification

   For invalid token errors, verify your bot token:

   # Test token directly
   curl "https://api.telegram.org/bot${TELEGRAM_BOT_TOKEN}/getMe"
   

Channel Keeps Retrying Forever

Symptom: High total_recoveries count, constant retry attempts visible in logs.

Investigation:

# Check supervision status
curl http://127.0.0.1:8765/health | jq '.channels'

# Look for:
# - High total_recoveries value
# - next_retry_at timestamp
# - last_error details

Resolution:

# Temporarily pause while investigating
praisonai gateway pause telegram

# Investigate root cause (network, DNS, etc.)
# Fix underlying issue

# Resume when ready
praisonai gateway resume telegram

---

Config Migration

When praisonai doctor --only gateway_config_migration reports Config can be migrated, your YAML uses a legacy shape that loads correctly but can be rewritten to the canonical format.

1. Detect migration opportunities

   praisonai doctor --only gateway_config_migration
   

   Example WARN output:

   ⚠ Gateway Config Migration [MEDIUM]
     Config can be migrated: 2 change(s)
     • telegram: allowed_users string → list migration available
     • Set telegram.group_policy to secure default 'mention_only'
   
2. Understand auto-normalisation

   At load time, GatewayConfigSchema already normalises legacy formats — your bot runs without rewriting the file. Persist the canonical form when you want the on-disk YAML to match what the schema produces.

3. Apply canonical YAML

   Legacy shape	Canonical form
   Top-level platform + token	channels: dict with one entry
   BotOS platforms: dict	channels: dict (same keys)
   allowed_users: "id1,id2" (string)	allowed_users: ["id1", "id2"] (list)

   See Gateway Config Migration for before/after YAML.

---

Environment Variables in Config

Gateway and bot configs support ${VAR} substitution in any string value. Resolution order:

1. Process environment variables
2. ~/.praisonai/.env (loaded automatically; override path with PRAISONAI_ENV_FILE)

# Verify all referenced variables resolve
praisonai doctor --only gateway_env_substitution

Unset variables cause validation to FAIL at load time and in the gateway_env_substitution doctor check.

---

Best Practices

Use praisonai gateway status first

Always run praisonai gateway status before diving into logs - it shows port, PID, and daemon state in one command.

Stop before restart

Always use praisonai gateway stop before restarting - this drains active sessions and prevents data loss.

Use onboard for fresh installs

When in doubt, run praisonai onboard - it handles daemon setup, credentials, and port configuration automatically.

Check logs for crash loops

If bot-stderr.log grows rapidly, you have a crash loop - stop the daemon, upgrade PraisonAI, then restart.

Related

Gateway CLI

Gateway command reference

Gateway Server

Gateway configuration and setup