> ## Documentation Index
> Fetch the complete documentation index at: https://praison.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Realtime Voice Agent

> Open a real WebSocket to OpenAI's Realtime API for live voice and text

`RealtimeAgent` opens a real WebSocket to OpenAI's Realtime API for bidirectional audio and text.

> [!NOTE]
>
> Looking for token-by-token **text** streaming (`agent.stream('Tell me a story')`)? That moved to [Streaming](https://praison.ai/docs/js/streaming.md). This page is the voice-first `RealtimeAgent`.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
graph LR
    subgraph "Realtime Voice"
        Mic[🎤 Audio / Text] --> A[🤖 RealtimeAgent]
        A --> WS[🔌 WebSocket]
        WS --> API[🧠 OpenAI Realtime]
        API --> Out[🔊 Audio / Text back]
    end

    classDef input fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef agent fill:#8B0000,stroke:#7C90A0,color:#fff
    classDef proc fill:#189AB4,stroke:#7C90A0,color:#fff
    classDef out fill:#10B981,stroke:#7C90A0,color:#fff

    class Mic input
    class A agent
    class WS,API proc
    class Out out
```

## Quick Start

**1. Connect and send text**

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({
  name: 'VoiceAssistant',
  llm: 'gpt-4o-realtime-preview',
  apiKey: process.env.OPENAI_API_KEY,
});

agent.onAudio((chunk) => playAudio(chunk));   // Uint8Array of decoded audio
agent.onMessage((text) => console.log(text)); // text deltas

await agent.connect();          // Real WebSocket — throws on auth/network failure
await agent.sendText('Hello!'); // Real API frames — throws if not connected
```

**2. Stream microphone audio**

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({
  instructions: 'You are a helpful voice assistant.',
  realtime: { voice: 'nova' },
  apiKey: process.env.OPENAI_API_KEY,
});

agent.onAudio((chunk) => playAudio(chunk));

await agent.connect();
agent.sendAudio(microphoneBuffer);  // Uint8Array | ArrayBuffer of PCM16
agent.commitAudio();                // ask the server to process the buffer
```

> [!WARNING]
>
> **What happens on connect failure.** `connect()` resolves only after the WebSocket handshake actually completes. A bad API key, an unreachable host, a close during the handshake, or a timeout all **reject** — the agent never reports a fake "connected" state. Wrap `connect()` in `try/catch` and expect an honest error.

***

## How It Works

Both audio and text turns travel over the same live socket.

```mermaid theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
sequenceDiagram
    participant User
    participant Agent as RealtimeAgent
    participant WS as WebSocket
    participant API as OpenAI Realtime

    User->>Agent: connect()
    Agent->>WS: open handshake (Authorization, OpenAI-Beta)
    WS->>API: wss://api.openai.com/v1/realtime?model=...
    API-->>WS: open
    WS-->>Agent: resolve connect()
    Agent->>API: session.update (voice, formats, VAD)

    User->>Agent: sendText("Hello") / sendAudio(buffer)
    Agent->>API: conversation.item.create + response.create
    API-->>Agent: response.text.delta / response.audio.delta
    Agent-->>User: onMessage(text) / onAudio(chunk)
```

| Behaviour | What actually happens |
| - | - |
| `connect()` | Resolves only on socket `open`; rejects on error, close, or timeout |
| `isConnected()` | Reads the live socket `readyState` — flips to `false` on a server-initiated close |
| `sendText()` / `sendAudio()` | Send real API frames; **throw** when not connected |
| Events | Every event is parsed off the wire; unknown server types pass through verbatim |

***

## Events

Register handlers by event type, or use `'*'` to receive every server event. Convenience callbacks mirror the Python agent.

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({ apiKey: process.env.OPENAI_API_KEY });

// Typed event handlers
agent.on('response.audio.delta', (event) => playAudio(event.data));
agent.on('response.text.delta', (event) => process.stdout.write(event.data.delta));

// Receive everything, including unknown server types (passed through verbatim)
agent.on('*', (event) => console.log(event.type));

// Python-parity convenience callbacks
agent.onMessage((text) => console.log(text));   // decoded text deltas
agent.onAudio((chunk) => playAudio(chunk));      // decoded audio bytes
agent.onError((err) => console.error(err));

await agent.connect();
```

Known event types include `session.created`, `session.updated`, `response.text.delta`, `response.audio.delta`, `response.done`, and `error`. Any other type the server sends is emitted untouched.

***

## Configuration Options

Pass `realtime` as a config object to tune voice and audio.

| Option | Type | Default | Description |
| - | - | - | - |
| `voice` | `'alloy' \| 'echo' \| 'fable' \| 'onyx' \| 'nova' \| 'shimmer'` | `'alloy'` | Response voice |
| `inputFormat` | `'pcm16' \| 'g711_ulaw' \| 'g711_alaw'` | `'pcm16'` | Audio input format |
| `outputFormat` | `'pcm16' \| 'g711_ulaw' \| 'g711_alaw'` | `'pcm16'` | Audio output format |
| `sampleRate` | `number` | `24000` | Local playback hint (not sent to the API) |
| `vadEnabled` | `boolean` | `true` | Server-side voice activity detection |
| `vadThreshold` | `number` | `0.5` | VAD sensitivity |
| `timeout` | `number` | `300` | Session timeout (seconds) |

Agent-level options: `llm` (default `gpt-4o-realtime-preview`), `apiKey` (falls back to `OPENAI_API_KEY`), `url` (endpoint override), `headers`, `webSocket` (inject a constructor), and `connectTimeoutMs` (default `30000`).

```typescript theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({
  name: 'VoiceAssistant',
  llm: 'gpt-4o-realtime-preview',
  apiKey: process.env.OPENAI_API_KEY,
  connectTimeoutMs: 10000,
  realtime: {
    voice: 'nova',
    inputFormat: 'pcm16',
    vadEnabled: true,
  },
});
```

> [card] [RealtimeAgent API Reference](https://praison.ai/docs/sdk/reference/typescript/classes/RealtimeAgent.md)
>
> Full class documentation

***

## Runtime Requirements

The WebSocket implementation is resolved at call time: an injected `{ webSocket }` constructor first, then `globalThis.WebSocket`, then the optional `ws` package.

| Runtime | How it connects |
| - | - |
| Node ≥ 22 | Global `WebSocket` with `{ headers }` for `Authorization` and `OpenAI-Beta` |
| Older Node | Install `ws` (`npm install ws`) — the fallback path picks it up |
| Browser | No handshake headers; falls back to OpenAI's documented subprotocol credentials |

> [!WARNING]
>
> The browser subprotocol path puts the API key in the page. Use an ephemeral key there. It is verified at wire level only.

If no implementation is available, `connect()` throws with the three ways to fix it (upgrade Node, install `ws`, or inject a constructor) — it never pretends to connect.

***

## Best Practices

> [accordion] Always try/catch connect()
>
> A bad key, a bad URL, or a timeout rejects. Handle the error instead of assuming a connection succeeded.

> [accordion] Send only after connect() resolves
>
> `sendText`, `sendAudio`, `commitAudio`, and `clearAudio` throw when disconnected. Await `connect()` first, and re-check `isConnected()` after a possible server close.

> [accordion] Handle unknown event types
>
> The server may send event types this SDK doesn't name. Register `'*'` so you see them all rather than dropping frames.

> [accordion] Use an ephemeral key in browsers
>
> Browsers cannot set handshake headers, so credentials travel as a subprotocol — visible in the page. Mint a short-lived key for that path.

> [!NOTE]
>
> `sampleRate` is a local playback hint and is not sent to the API. The low-level `SQLiteAdapter` in `praisonai/db/sqlite` is unrelated to this agent and still degrades to a `Map` for legacy callers.

***

## Related

> [card] [Streaming](https://praison.ai/docs/js/streaming.md)
>
> Token-by-token text streaming

> [card] [Voice](https://praison.ai/docs/js/voice.md)
>
> Voice interactions

> [card] [Audio](https://praison.ai/docs/js/audio.md)
>
> Audio input and output

> [card] [Agent](https://praison.ai/docs/js/agent.md)
>
> Create agents


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.