Voice Reply (Text-to-Speech)

Speak the agent’s reply back as a native voice note on Telegram, Slack, WhatsApp, and Discord

Speak the agent’s plain-text reply back as a native voice note — the outbound counterpart to inbound speech-to-text, off by default and opt-in per channel.

from praisonaiagents import Agent

# 1. Author the agent exactly as before — voice is a transport concern.
agent = Agent(
    name="assistant",
    instructions="Be helpful and concise.",
)

# 2. Enable voice replies on your Telegram channel in gateway.yaml:
#
#    channels:
#      telegram:
#        platform: telegram
#        token: ${TELEGRAM_BOT_TOKEN}
#        stt:   true                    # user speaks in
#        voice: { mode: match_inbound } # bot speaks back when they do
#
# 3. Send a voice note to the bot on Telegram — you get a spoken reply.
#    Send a text message — you get a text reply.

One YAML line closes the voice loop — the bot listens and speaks.

Voice reply mirrors inbound Voice Notes (Speech-to-Text), but it is off by default. Set voice.mode: match_inbound for the intuitive “speak in / speak out” symmetry.

Quick Start

1. Level 1 — Bool shorthand

   voice: true turns replies into voice notes for every message (mode: always).

   channels:
     telegram:
       platform: telegram
       token: ${TELEGRAM_BOT_TOKEN}
       voice: true    # every reply is also spoken
   
2. Level 2 — Mode shorthand

   A bare mode implies enabled: true. Use match_inbound to speak back only when the user sent a voice memo.

   channels:
     telegram:
       platform: telegram
       token: ${TELEGRAM_BOT_TOKEN}
       voice:
         mode: match_inbound    # speak back only when spoken to
   
3. Level 3 — Dict

   Tune the voice, model, speed, format, and length cap.

   channels:
     telegram:
       platform: telegram
       token: ${TELEGRAM_BOT_TOKEN}
       voice:
         enabled: true
         mode: match_inbound
         model: openai/tts-1
         voice: nova
         speed: 1.0
         format: ogg
         max_chars: 4000
   
4. Level 4 — TtsConfigSchema

   Build the channel config in Python with the validated schema.

   from praisonai_bot.bots._config_schema import (
       ChannelConfigSchema,
       TtsConfigSchema,
   )
   
   channel = ChannelConfigSchema(
       platform="telegram",
       token="${TELEGRAM_BOT_TOKEN}",
       voice=TtsConfigSchema(
           enabled=True,
           mode="match_inbound",
           model="openai/tts-1",
           voice="nova",
           speed=1.0,
           format="ogg",
           max_chars=4000,
       ),
   )
   

The tts: block is an accepted alias for voice: anywhere. voice: true is shorthand for {enabled: true, mode: always}.

---

How It Works

The gateway synthesises the reply after the text is already sent, so voice is always a bonus — never a blocker.

Step	What happens
Reply	The agent returns plain text; the text reply is delivered first, every time.
Gate	should_voice_reply checks enabled + mode (and, for match_inbound, whether the inbound was a voice memo).
Synthesise	synthesize_voice_reply calls tts_tool → AudioAgent.speech (default openai/tts-1).
Deliver	The adapter sends the file as a native voice note (Telegram send_voice).
Degrade	Any skip or failure logs and is dropped — the text reply already reached the user.

Graceful degradation

Voice reply is best-effort — the text reply is always delivered first.

• Empty / whitespace text → skipped silently.
• Reply longer than max_chars → skipped, INFO log entry.
• TTS tool import fails (missing openai/litellm deps) → skipped, WARNING log entry.
• Synthesis error → skipped, ERROR log entry.
• Adapter send_voice fails → logged; the text reply already went out.

---

Configuration Options

Fields from TtsConfig / TtsConfigSchema.

Option	Type	Default	Description
enabled	bool	False	Master switch. Off by default (opt-in) — the mirror of STT’s on-by-default inbound transcription.
mode	str	"off"	Delivery mode: "off" | "always" | "match_inbound". Ignored when enabled is False.
model	Optional[str]	None	Optional TTS model override. None uses openai/tts-1.
voice	Optional[str]	None	Optional voice name ("alloy", "echo", "fable", "onyx", "nova", "shimmer").
speed	Optional[float]	None	Speaking-rate multiplier (0.25–4.0). Provider default when None.
format	str	"ogg"	Output audio format. "ogg" / "opus" are the voice-note native formats.
max_chars	int	4000	Skip TTS for replies longer than this. 0 disables the cap.
stream	bool	False	Opt-in sentence-level streaming. When True, the reply is split into sentences and each is synthesised & delivered as it becomes available — time-to-first-audio drops from full-response latency to first-sentence latency. Falls back to a single whole-clip synth when off (default), when the reply is a single sentence, or when the reply is over max_chars.

Bool shorthand

channels:
  telegram:
    voice: true    # equivalent to {enabled: true, mode: always}

Resolution order

The effective policy is resolved from, in order:

1. config.metadata["voice"] (operator override),
2. config.metadata["tts"] (alias),
3. a direct config.voice attribute (schema-backed configs),
4. a direct config.tts attribute, then
5. the off-by-default TtsConfig().

---

Choosing a Mode

Pick the mode that matches how chatty your users want the bot to be.

Mode	When to use
off	Default. Never reply in voice.
match_inbound	Recommended. Reply in voice only when the user sent a voice memo — the natural mirror of STT.
always	Every reply is also a voice note — best for voice-first or accessibility bots.

mode: match-inbound (hyphen) normalises to match_inbound automatically.

---

Sentence-Level Streaming

Streaming TTS synthesises and delivers your reply sentence-by-sentence, so the user hears the first clause while the rest is still being spoken.

With stream: false (default) the whole reply is synthesised into one clip before anything plays — the user waits for full-response latency. With stream: true time-to-first-audio drops to first-sentence latency.

YAML

channels:
  telegram:
    platform: telegram
    token: ${TELEGRAM_BOT_TOKEN}
    voice:
      mode: match_inbound
      stream: true      # NEW: synthesise & deliver sentence-by-sentence

When streaming kicks in

Streaming is used only when all three preconditions hold — otherwise the whole-file path runs.

• stream: true is set.
• The full reply is under max_chars (the cap is checked against the whole reply before the per-sentence loop, so long replies can’t sneak through as many under-cap clauses).
• The reply splits into more than one sentence. A single-sentence reply falls through to the whole-file path — streaming buys nothing there.

Failure semantics

Failures are audible-output-aware, so the user never hears a reply restart.

• Nothing played yet + the first clip fails → falls back to whole-file synth on the full text.
• Some audio already played + a clause fails → skip that clause and continue; never replays from the beginning.

When to enable

Flip stream: true when replies are usually multi-sentence and users are waiting for audio.

---

Per-Platform YAML

Telegram is wired today; other adapters share the same _tts helper and can adopt it without further core changes.

Telegram

channels:
  telegram:
    platform: telegram
    token: ${TELEGRAM_BOT_TOKEN}
    stt: true                  # user speaks in
    voice:
      mode: match_inbound      # bot speaks back when they do
      voice: nova
      stream: true             # sentence-level delivery (opt-in)

---

Platform Support

Platform	Voice reply status	Notes
Telegram	✅ Wired	Reference consumer — delivers via send_voice. Optional sentence-level streaming (stream: true) delivers a clip per sentence.
Slack	🔜 Adapter can adopt	Same shared _tts helper; adapter update needed.
WhatsApp	🔜 Adapter can adopt	Same shared _tts helper; adapter update needed.
Discord	🔜 Adapter can adopt	Same shared _tts helper; adapter update needed.

The [[audio_as_voice]] manual escape hatch continues to work for pre-rendered audio on Telegram.

---

User Interaction Flow

A voice-in / voice-out conversation on Telegram with mode: match_inbound.

1. The user taps 🎙️ and records a voice note on Telegram.
2. Inbound STT transcribes it → the agent processes the text → produces a plain-text reply.
3. The gateway sees voice.mode: match_inbound and that the inbound was a voice memo → synthesises the reply with openai/tts-1.
4. The user receives both: the text reply and a native voice note they can play right in the chat.
5. If the user next sends a text message, they get only a text reply — no unnecessary audio.
6. When stream: true is set, the user hears the first sentence as soon as it is synthesised while the rest of the reply is still being spoken.

---

Best Practices

Prefer match_inbound over always

match_inbound speaks back only when the user spoke first — the symmetry users expect. Reserve always for voice-first or accessibility bots.

voice:
  mode: match_inbound

Cap length for chatty agents

Set max_chars low so long analytical replies don’t become multi-minute audio clips. The text reply still carries the full answer.

voice:
  mode: match_inbound
  max_chars: 600

Ship your TTS provider's API key

openai/tts-1 is the default, so set OPENAI_API_KEY. Any LiteLLM-supported TTS model works via the model: field.

Trust the graceful-degradation contract

Voice reply is best-effort — the plain-text reply is always delivered first, so users never lose a message even when synthesis fails.

Stream long replies for lower time-to-first-audio

Set stream: true when replies are typically more than one sentence and users are waiting for audio. Time-to-first-audio drops from full-response latency to first-sentence latency, and the whole-file path is still used automatically for single-sentence replies or replies over max_chars.

voice:
  mode: match_inbound
  stream: true

Keep the manual escape hatch for pre-rendered audio

The [[audio_as_voice]] marker still works on Telegram for audio you’ve already produced — e.g. a recording pulled from a knowledge base.

---

Related

Voice Notes (Speech-to-Text)

The inbound counterpart — transcribe voice notes into text for the agent.

Audio Tools

tts_tool, stt_tool, and AudioAgent.speech.

Voice Notes

The inbound voice-notes doc for gateway bots.

Streaming Replies

The text-streaming counterpart — stream the reply text into chat as it is generated.

Gateway

The overall gateway pattern — voice reply is one of several channel features.