Gateway API Endpoints

Serve OpenAI-compatible and MCP HTTP endpoints from the running gateway

Turn the running gateway into an OpenAI-compatible and MCP endpoint so SDK clients and MCP tools reach the same live agents as chat users.

One process, one agent state, three protocols. This is different from praisonai serve openai, which runs a separate standalone OpenAI-only process.

Quick Start

1. Enable in gateway.yaml

   Add the api block to your gateway config:

   gateway:
     api:
       openai: true
       mcp: true
   
2. Start with CLI flags

   Enable the same surfaces from the command line:

   praisonai gateway start --config gateway.yaml --openai-api --mcp
   
3. Enable in Python

   Pass constructor flags to the gateway:

   from praisonai.gateway import WebSocketGateway
   
   WebSocketGateway(config=cfg, openai_api=True, mcp=True).start()
   

---

How It Works

Each API request dispatches into the gateway’s own registered agents, sharing the same session store and admission gate as chat users.

Step	What happens
Resolve agent	The model field selects a registered agent; falls back to the first agent
Reuse session	Each caller gets a stable session keyed by OpenAI-Session / X-Session-Id header or bearer token
Admit turn	The turn passes through the same admission gate as chat users
Respond	The reply is returned in OpenAI or MCP shape

---

Endpoints Exposed

Surface	Method + Path	Notes
OpenAI chat	POST /v1/chat/completions	SSE streaming supported (stream: true); real usage returned; opt-in streamed usage via stream_options.include_usage; opt-in token-level delta.content frames via gateway.api.stream (default off keeps single-chunk buffered behaviour)
OpenAI responses (sync)	POST /v1/responses	Default — blocks for the whole turn; accepts string or messages-style input; returns status: "completed" with usage in Responses-native shape
OpenAI responses (async)	POST /v1/responses with background: true or store: true	Returns 202 immediately with status: "queued" and a persisted id
OpenAI response retrieve	GET /v1/responses/{id}	Fetches the current status + output of a background/stored run
OpenAI response cancel	POST /v1/responses/{id}/cancel	Stops an in-flight background run cooperatively
OpenAI models	GET /v1/models	Lists gateway-registered agent IDs
MCP JSON-RPC	POST /mcp	Methods: initialize, tools/list, tools/call

Call the OpenAI surface with any standard client:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="gateway-token")
reply = client.chat.completions.create(
    model="assistant",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

---

Background Runs

Submit long agent turns in the background, retrieve the result by id, and cancel in-flight runs — over the same OpenAI-compatible surface.

The synchronous path is unchanged and remains the default — omit background and store to keep the exact previous behaviour.

Which mode fits

Quick Start

1. Submit in the background

   import requests
   
   resp = requests.post(
       "http://127.0.0.1:8765/v1/responses",
       headers={"Authorization": "Bearer gateway-token"},
       json={"model": "assistant", "input": "Write me a long essay", "background": True},
   )
   run_id = resp.json()["id"]
   print(resp.status_code, resp.json()["status"])   # 202 queued
   
2. Poll until terminal

   import time, requests
   
   while True:
       got = requests.get(
           f"http://127.0.0.1:8765/v1/responses/{run_id}",
           headers={"Authorization": "Bearer gateway-token"},
       ).json()
       if got["status"] in ("completed", "cancelled", "failed"):
           break
       time.sleep(2)
   print(got["output_text"])
   
3. Cancel if needed

   import requests
   
   requests.post(
       f"http://127.0.0.1:8765/v1/responses/{run_id}/cancel",
       headers={"Authorization": "Bearer gateway-token"},
   )
   

User Flow

Status Vocabulary

Status	Meaning
queued	Submitted; the driving task has not started yet (immediately after 202)
in_progress	Task is running the agent turn
completed	Turn finished; output, output_text, usage are populated
cancelled	The run’s cancel route (or task cancel) fired before completion
failed	The turn raised an exception, or the admission gate rejected it (error.message explains)

A completed response always carries an assistant message (even with empty text), so a client reading output[0] keeps working. Polling the same completed response returns the same output[0].id each time.

Response Object

{
  "id": "resp-<hex>",
  "object": "response",
  "created_at": 1727612345,
  "model": "<agent_id>",
  "status": "queued|in_progress|completed|cancelled|failed",
  "output": [
    { "id": "msg-<hex>", "type": "message", "role": "assistant",
      "content": [{"type": "output_text", "text": "..."}] }
  ],
  "output_text": "...",
  "usage": { "input_tokens": 0, "output_tokens": 0, "total_tokens": 0 },
  "error": { "message": "..." }
}

The error block appears only when status is failed. A GET on an unknown id returns 404 with {"error": {"type": "invalid_request_error", "message": "No such response: resp-…"}}.

When the caller uses a stable identity (a bearer token, or an OpenAI-Session / X-Session-Id header), only that caller can retrieve or cancel their own background runs. A different token receives 404 (not 403, to avoid leaking id existence). Anonymous callers on auth-disabled gateways are looked up by id alone — the same open posture that applies to every /v1/* route without auth.

Background responses live in a bounded in-process store (cap: 1024 entries). When full, the oldest terminal entry (completed/cancelled/failed) is evicted first; in-flight runs are never evicted. Results do not survive a gateway restart — this is intentional for now; durable cross-restart persistence is tracked separately.

Both new routes go through the same _api_guarded wrapper as POST /v1/responses (gated by gateway.auth_token) and only mount when gateway.api.openai = true.

---

Safe Retries (Idempotency-Key)

Send the same Idempotency-Key header on a retry and the gateway returns the first run instead of starting a second — no duplicate LLM spend, no duplicate tool calls.

Opt-in, backward-compatible: no header → unchanged behaviour (every POST starts a fresh run, as before).

Quick Start

1. Send a key with /v1/responses

   Any UUID-shaped string works. Reuse the exact same value on every retry of the same logical request.

   import uuid, requests
   
   key = str(uuid.uuid4())
   body = {"model": "assistant", "input": "Write a short poem"}
   headers = {
       "Authorization": "Bearer gateway-token",
       "Idempotency-Key": key,
   }
   
   resp = requests.post("http://127.0.0.1:8765/v1/responses", json=body, headers=headers)
   print(resp.json()["output_text"])
   
2. Retry safely with the same key

   If the first POST timed out or your load balancer re-sent it, replay with the same key and body — you’ll get the first run’s answer back, not a second run.

   # Same key + same body → first run's response, no second agent run
   retry = requests.post("http://127.0.0.1:8765/v1/responses", json=body, headers=headers)
   assert retry.json()["id"] == resp.json()["id"]  # same run id
   
3. Works on /v1/chat/completions too

   import uuid, requests
   
   key = str(uuid.uuid4())
   body = {
       "model": "assistant",
       "messages": [{"role": "user", "content": "Hello"}],
   }
   headers = {"Authorization": "Bearer gateway-token", "Idempotency-Key": key}
   
   requests.post("http://127.0.0.1:8765/v1/chat/completions", json=body, headers=headers)
   

What happens on each case

Case	Behaviour
No header	Unchanged — every POST starts a fresh run.
First use of a key	Run is created; the key is recorded with a fingerprint of the request body.
Retry, same key + equivalent body	The original run is returned. For /v1/responses background runs, you see the run’s current state (completed with the real output_text once it finishes), not the stale queued snapshot.
Same key + materially different body	409 Conflict with error.type: "invalid_request_error" — the gateway refuses to silently honour either request.
Concurrent retry while the first run is still working	The second POST waits for the first run and gets the same response — not a second run.

User Flow

Scoping and limits

Rule	Why
Per route	A key reused across /v1/responses and /v1/chat/completions never replays one route’s response shape on the other — a chat.completion is never returned for a response request.
Per caller	Keys scoped by bearer token or OpenAI-Session / X-Session-Id header. Two different callers using the same key get independent runs on a shared gateway. Anonymous callers on auth-disabled gateways share one “anon” scope — the client’s own key is the handle.
Streaming excluded	stream: true chat completions bypass the idempotency check — streamed responses are not replayable. Use the buffered (non-stream) path when you need safe retries.
In-process, bounded	Up to 1024 keys in RAM; evicts only entries whose run is terminal, so a long-running run is never evicted mid-flight. Does not survive a gateway restart.
No request bodies stored	The gateway stores a SHA-256 fingerprint of the canonicalised body plus the public run id/status — never your request body or credentials.

The feature activates automatically whenever gateway.api.openai = true. There is no YAML config knob, no CLI flag, and no new dependency.

Because the key store is in-process and bounded (cap 1024), a flood of unkeyed background submissions can evict older keyed entries once their runs reach a terminal state. A retry of a key whose underlying run was evicted still replays the stored terminal result (the gateway refreshes the committed payload at completion) — it never replays the stale queued snapshot.

---

Token Usage

Every response reports real per-turn token counts from the agent’s LLM instance.

Each surface reports usage in its own OpenAI-native shape:

Surface	usage shape
/v1/chat/completions	{prompt_tokens, completion_tokens, total_tokens} (OpenAI Chat Completions)
/v1/responses	{input_tokens, output_tokens, total_tokens} (OpenAI Responses-native)

When an agent exposes no LLM metrics, usage falls back to zeros in the correct spec shape, so clients never need to special-case a missing block.

Streamed usage

Opt in with stream_options.include_usage: true to receive a streamed usage chunk — the same contract OpenAI’s own API uses.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="gateway-token")
stream = client.chat.completions.create(
    model="assistant",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True,
    stream_options={"include_usage": True},
)
for chunk in stream:
    if chunk.usage:
        print(chunk.usage)  # final chunk: real prompt/completion/total tokens

The usage-only chunk is emitted right before data: [DONE]. Without include_usage, the stream frames are unchanged and no extra chunk is sent. This is orthogonal to token-level streaming below — it works with or without it.

Token-level streaming (opt-in)

Enable gateway.api.stream to emit each DELTA_TEXT / FIRST_TOKEN piece as its own chat.completion.chunk with delta.content, so time-to-first-content matches token latency instead of full-turn latency.

Default is off: without opt-in the SSE surface is byte-for-byte the previous single buffered chunk, so existing clients keep working unchanged.

Enable it in gateway.yaml:

gateway:
  api:
    openai: true
    stream: true   # opt-in token-level SSE on /v1/chat/completions

Or in Python:

from praisonai.gateway import WebSocketGateway
from praisonaiagents.gateway import ApiConfig

WebSocketGateway(
    config=cfg,
    api=ApiConfig(openai=True, stream=True),
).start()

Consume it from any standard OpenAI client — no client-side change needed:

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8765/v1", api_key="gateway-token")
stream = client.chat.completions.create(
    model="assistant",
    messages=[{"role": "user", "content": "Write a haiku about the sea"}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta
    if delta.content:
        print(delta.content, end="", flush=True)  # arrives token-by-token

Behaviour guarantees:

Guarantee	What it means
Same frame envelope	The role prelude, finish_reason: "stop" terminator, stream_options.include_usage, and [DONE] sentinel are all preserved.
No-delta fallback	If the agent emits no token deltas, the buffered reply is still delivered as one delta.content frame — nothing is lost, even for an empty reply.
Reasoning filtered	Thinking/reasoning deltas are excluded from the answer stream.
Concurrent isolation	A per-turn guard ensures two callers sharing one agent never receive each other’s tokens.

---

Configuration Options

The gateway.api block maps to the ApiConfig dataclass. All three flags are opt-in; when openai and mcp are both False (default), no extra routes are mounted.

Option	Type	Default	Description
openai	bool	False	Serve /v1/chat/completions, /v1/responses, /v1/models backed by the gateway’s live agents and sessions.
mcp	bool	False	Serve an MCP JSON-RPC endpoint at /mcp exposing registered agents as callable tools.
stream	bool	False	Emit true token-level delta.content frames on /v1/chat/completions SSE by wiring the agent’s stream_emitter through the gateway hot path. Off by default: streaming responses stay byte-for-byte the buffered single-chunk path.

ApiConfig also exposes an enabled property (true if either surface is on), plus to_dict() and from_dict().

Full API surface

---

How Auth Works

Every API route is protected by the same gateway.auth_token as /info and /metrics.

The /info endpoint advertises which surfaces are enabled in its api field, so clients can introspect a running gateway before connecting.

GET /info also returns scopes, the caller’s effective operator scopes on this request, resolved from the bearer token against the auth_scopes policy. With no policy, every authenticated caller gets every scope.

{
  "version": "2.3.4",
  "protocol_version": 7,
  "sessions": 7,
  "clients": 3,
  "api": {},
  "scopes": ["read", "write"]
}

praisonai gateway status prints this as an Access: line — see Gateway CLI → status and Operator Scopes.

---

When to Use vs praisonai serve openai

Use the gateway api: block when you want SDK clients and MCP tools to share live agents and sessions with chat users. Use praisonai serve openai when you want a standalone, lightweight OpenAI-only process with no gateway state. See OpenAI-Compatible Server.

Need	Choose
SDK clients share chat users’ live sessions	Gateway api: block
Standalone OpenAI-only endpoint	praisonai serve openai
Expose gateway agents as MCP tools	Gateway api.mcp

---

Best Practices

Keep surfaces off unless needed

Leave openai: false and mcp: false (the defaults) unless a client needs them. Disabled surfaces mount no routes and leave the gateway unchanged.

Protect with auth_token in production

Every /v1/* and /mcp route uses the same gateway.auth_token. Set a strong token when binding to any non-loopback interface.

Pin conversations with a session header

Pass an OpenAI-Session or X-Session-Id header to reuse one agent session across calls. Without it, stateless callers get a fresh session per request.

Introspect surfaces with /info

GET /info returns an api field listing enabled surfaces, so tooling can confirm what a gateway exposes before dispatching.

Enable streamed usage for cost accounting

Pass stream_options={"include_usage": True} when using stream=True if your client tracks cost per response. The extra chunk arrives right before [DONE] and carries real per-turn totals.

Usage is per-turn, even under concurrent sessions

The gateway snapshots the agent’s per-turn token metrics in the same execution context that produced the reply, so a usage field returned to one caller cannot be corrupted by a concurrent turn on the same shared agent. No configuration required.

Run long turns in the background

Any turn that could exceed the client’s HTTP timeout is a candidate. POST /v1/responses with background: true returns a persisted id in ~milliseconds; poll GET /v1/responses/{id} at a comfortable interval (2–5s) until status is terminal. If the caller loses connectivity, the run keeps going and the result is still retrievable by id (subject to the in-process store cap).

Set a bearer token to scope background runs

Owner-scoped retrieval only kicks in when the caller has a stable identity. Anonymous callers on auth-disabled gateways are looked up by id alone. Set gateway.auth_token in any deployment where multiple clients share the same gateway.

Cancel is cooperative

/cancel fires the same interrupt seam as the WebSocket /stop path — the agent checks it at its own checkpoints and stops between LLM calls / tool calls. A tight, non-yielding loop inside a tool won’t stop until it yields; the task-cancel fallback covers most cases but not synchronous CPU-bound work.

Set an Idempotency-Key for safe retries

Any production client should set Idempotency-Key: <uuid> on every POST /v1/responses and POST /v1/chat/completions. The gateway dedupes the retry at the API boundary, so a timeout or load-balancer re-send never starts a second agent run or doubles your LLM spend. Reuse the same key on every retry of the same logical request. Applies to the buffered (non-stream) path only — streaming responses are not replayable.

---

Related

Gateway

Gateway architecture and YAML configuration

OpenAI-Compatible Server

Standalone OpenAI-only server process

MCP Integration

Model Context Protocol servers and clients

Gateway CLI

CLI commands for managing the gateway