Realtime Voice Agent

Open a real WebSocket to OpenAI’s Realtime API for live voice and text

RealtimeAgent opens a real WebSocket to OpenAI’s Realtime API for bidirectional audio and text.

Looking for token-by-token text streaming (agent.stream('Tell me a story'))? That moved to Streaming. This page is the voice-first RealtimeAgent.

Quick Start

1. Connect and send text

   import { RealtimeAgent } from 'praisonai';
   
   const agent = new RealtimeAgent({
     name: 'VoiceAssistant',
     llm: 'gpt-4o-realtime-preview',
     apiKey: process.env.OPENAI_API_KEY,
   });
   
   agent.onAudio((chunk) => playAudio(chunk));   // Uint8Array of decoded audio
   agent.onMessage((text) => console.log(text)); // text deltas
   
   await agent.connect();          // Real WebSocket — throws on auth/network failure
   await agent.sendText('Hello!'); // Real API frames — throws if not connected
   
2. Stream microphone audio

   import { RealtimeAgent } from 'praisonai';
   
   const agent = new RealtimeAgent({
     instructions: 'You are a helpful voice assistant.',
     realtime: { voice: 'nova' },
     apiKey: process.env.OPENAI_API_KEY,
   });
   
   agent.onAudio((chunk) => playAudio(chunk));
   
   await agent.connect();
   agent.sendAudio(microphoneBuffer);  // Uint8Array | ArrayBuffer of PCM16
   agent.commitAudio();                // ask the server to process the buffer
   

What happens on connect failure. connect() resolves only after the WebSocket handshake actually completes. A bad API key, an unreachable host, a close during the handshake, or a timeout all reject — the agent never reports a fake “connected” state. Wrap connect() in try/catch and expect an honest error.

---

How It Works

Both audio and text turns travel over the same live socket.

Behaviour	What actually happens
connect()	Resolves only on socket open; rejects on error, close, or timeout
isConnected()	Reads the live socket readyState — flips to false on a server-initiated close
sendText() / sendAudio()	Send real API frames; throw when not connected
Events	Every event is parsed off the wire; unknown server types pass through verbatim

---

Events

Register handlers by event type, or use '*' to receive every server event. Convenience callbacks mirror the Python agent.

import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({ apiKey: process.env.OPENAI_API_KEY });

// Typed event handlers
agent.on('response.audio.delta', (event) => playAudio(event.data));
agent.on('response.text.delta', (event) => process.stdout.write(event.data.delta));

// Receive everything, including unknown server types (passed through verbatim)
agent.on('*', (event) => console.log(event.type));

// Python-parity convenience callbacks
agent.onMessage((text) => console.log(text));   // decoded text deltas
agent.onAudio((chunk) => playAudio(chunk));      // decoded audio bytes
agent.onError((err) => console.error(err));

await agent.connect();

Known event types include session.created, session.updated, response.text.delta, response.audio.delta, response.done, and error. Any other type the server sends is emitted untouched.

---

Configuration Options

Pass realtime as a config object to tune voice and audio.

Option	Type	Default	Description
voice	'alloy' | 'echo' | 'fable' | 'onyx' | 'nova' | 'shimmer'	'alloy'	Response voice
inputFormat	'pcm16' | 'g711_ulaw' | 'g711_alaw'	'pcm16'	Audio input format
outputFormat	'pcm16' | 'g711_ulaw' | 'g711_alaw'	'pcm16'	Audio output format
sampleRate	number	24000	Local playback hint (not sent to the API)
vadEnabled	boolean	true	Server-side voice activity detection
vadThreshold	number	0.5	VAD sensitivity
timeout	number	300	Session timeout (seconds)

Agent-level options: llm (default gpt-4o-realtime-preview), apiKey (falls back to OPENAI_API_KEY), url (endpoint override), headers, webSocket (inject a constructor), and connectTimeoutMs (default 30000).

import { RealtimeAgent } from 'praisonai';

const agent = new RealtimeAgent({
  name: 'VoiceAssistant',
  llm: 'gpt-4o-realtime-preview',
  apiKey: process.env.OPENAI_API_KEY,
  connectTimeoutMs: 10000,
  realtime: {
    voice: 'nova',
    inputFormat: 'pcm16',
    vadEnabled: true,
  },
});

RealtimeAgent API Reference

Full class documentation

---

Runtime Requirements

The WebSocket implementation is resolved at call time: an injected { webSocket } constructor first, then globalThis.WebSocket, then the optional ws package.

Runtime	How it connects
Node ≥ 22	Global WebSocket with { headers } for Authorization and OpenAI-Beta
Older Node	Install ws (npm install ws) — the fallback path picks it up
Browser	No handshake headers; falls back to OpenAI’s documented subprotocol credentials

The browser subprotocol path puts the API key in the page. Use an ephemeral key there. It is verified at wire level only.

If no implementation is available, connect() throws with the three ways to fix it (upgrade Node, install ws, or inject a constructor) — it never pretends to connect.

---

Best Practices

Always try/catch connect()

A bad key, a bad URL, or a timeout rejects. Handle the error instead of assuming a connection succeeded.

Send only after connect() resolves

sendText, sendAudio, commitAudio, and clearAudio throw when disconnected. Await connect() first, and re-check isConnected() after a possible server close.

Handle unknown event types

The server may send event types this SDK doesn’t name. Register '*' so you see them all rather than dropping frames.

Use an ephemeral key in browsers

Browsers cannot set handshake headers, so credentials travel as a subprotocol — visible in the page. Mint a short-lived key for that path.

sampleRate is a local playback hint and is not sent to the API. The low-level SQLiteAdapter in praisonai/db/sqlite is unrelated to this agent and still degrades to a Map for legacy callers.

---

Related

Streaming

Token-by-token text streaming

Voice

Voice interactions

Audio

Audio input and output

Agent

Create agents