How to Implement Real-Time Speech Synthesis for AI Agents: A Complete Guide

Real-time speech synthesis for AI agents relies on a cascaded pipeline that captures microphone audio, transcribes it via ASR, processes it through an LLM, and streams synthesized speech back through a WebSocket connection with Voice Activity Detection (VAD) to minimize latency.

The bojieli/ai-agent-book repository demonstrates production-ready patterns for implementing real-time speech synthesis for agents using a cascaded pipeline architecture. This Node.js and Next.js implementation combines Silero VAD for end-of-speech detection, provider-agnostic ASR and TTS adapters, and WebSocket streaming to deliver conversational AI with voice interfaces that feel instantaneous.

Architecture Overview

The repository implements a full-stack solution split between client-side audio capture and server-side processing orchestration.

Frontend Audio Capture (Next.js)

The frontend application handles live microphone input using the Web Audio API. It captures raw audio samples and streams them over a WebSocket connection to the backend server, then plays back synthesized audio responses as they arrive.

Backend Processing Pipeline (Node.js)

The backend orchestrates the synthesis workflow through four sequential stages managed by backend/server.js. This central WebSocket server coordinates Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), Large Language Model (LLM) inference, and Text-to-Speech (TTS) generation.

The Real-Time Pipeline Components

Voice Activity Detection with Silero VAD

The backend/utils/vad.js module implements Silero VAD (Voice Activity Detection) to identify when a user stops speaking. Instead of processing continuous audio streams, the system waits for approximately 500ms of silence to mark the end of a speech turn.

This approach reduces unnecessary API calls and ensures the ASR service receives complete utterances rather than partial fragments, which improves transcription accuracy while maintaining low latency.

Provider-Agnostic ASR

The backend/utils/providers/asrProviders.js file implements adapter patterns for multiple ASR services. The system currently supports OpenAI Whisper and Siliconflow SenseVoice, allowing you to switch transcription providers through configuration changes rather than code rewrites.

Provider-Agnostic LLM Routing

Similarly, backend/utils/providers/llmProviders.js routes transcribed text to various LLM backends including OpenAI, OpenRouter, and ARK. This abstraction layer ensures the speech synthesis pipeline remains compatible with evolving language model APIs.

Low-Latency TTS Streaming

The backend/utils/providers/ttsProviders.js handles speech synthesis, supporting OpenAI's gpt-4o-mini-tts model and fallback tts-1 endpoints. The critical optimization occurs in backend/server.js, where audio bytes stream directly to the client via the established WebSocket connection as soon as the LLM response completes, eliminating wait times associated with batch processing.

Achieving Low Latency

Real-time performance depends on two specific architectural techniques implemented in the source code:

Streaming VAD → Chunked ASR: The VAD system buffers incoming audio until silence is detected, then sends the complete utterance to the ASR service. This batching approach, managed in backend/utils/vad.js, minimizes API round trips while ensuring the transcription context remains intact.

Concurrent TTS Streaming: Rather than waiting for the entire LLM response to finish, the backend initiates TTS requests as soon as the language model begins generating text. The synthesized audio chunks flow through the same WebSocket channel established in backend/server.js, enabling near-instant playback on the client side.

Advanced Control with Controllable-TTS

For sophisticated voice output beyond standard synthesis, the repository includes a controllable-TTS experiment that enables emotional and stylistic variation.

The system uses special markup markers (e.g., [EMO:happy], [SPEED:fast]) embedded in LLM responses. The markup.py parser extracts these instructions and maps them to reference voice profiles defined in voice_library.py, then passes them as instructions to OpenAI's TTS model. This allows agents to dynamically adjust tone, pacing, and emotional expression based on conversational context.

Implementation Guide

Backend Setup

Install and configure the Node.js WebSocket server that powers the real-time pipeline:

cd backend
npm install
cp config.js.example config.js   # edit provider choices if needed

npm run start                     # starts WebSocket server on port 8848

Requirements include Node.js ≥16, ffmpeg, and valid API keys for your chosen providers.

Frontend Setup

Launch the Next.js interface that handles microphone input and audio playback:

cd ../frontend
npm install
cp .env.example .env   # set WEBSOCKET_PORT=8848

npm run dev            # opens http://localhost:3000

Controllable-TTS Demo

Test the marker-driven synthesis system:

cd ../chapter9/controllable-tts
pip install -r requirements.txt
cp env.example .env
python demo.py --quick            # generates A/B/C comparison mp3 files

python demo.py --text "[EMO:happy][SPEED:fast]Thanks!" -o output.mp3

Streaming Speech Simulation

Evaluate the chunked ASR implementation without full integration:

cd ../chapter9/streaming-speech
pip install -r requirements.txt
cp env.example .env
python demo.py --quick            # runs chunked Whisper recognition

python demo.py --offline          # offline mode (no API key needed)

Summary

• Real-time speech synthesis for agents requires a cascaded pipeline architecture connecting audio input, VAD, ASR, LLM, and TTS components with minimal latency between stages.
• The backend/server.js WebSocket implementation enables streaming audio delivery, while backend/utils/vad.js provides Silero-based Voice Activity Detection to optimize transcription timing.
• Provider-agnostic adapters in asrProviders.js, llmProviders.js, and ttsProviders.js allow flexible integration with OpenAI, Siliconflow, OpenRouter, and other services without architectural changes.
• The controllable-TTS system uses markup.py and voice_library.py to parse emotional markers and generate dynamic speech variations through OpenAI's instruction-capable TTS models.

Frequently Asked Questions

What is real-time speech synthesis for AI agents?

Real-time speech synthesis for AI agents refers to systems that convert text responses into spoken audio with minimal perceptible delay, typically achieved through streaming architectures that process voice input and generate voice output concurrently rather than waiting for complete utterances or responses.

How does VAD reduce latency in speech synthesis pipelines?

Voice Activity Detection reduces latency by identifying natural speech boundaries through backend/utils/vad.js, which detects approximately 500ms of silence to trigger ASR submission. This prevents premature API calls on incomplete phrases and ensures the ASR service receives full contextual utterances, avoiding the errors and reprocessing delays associated with partial transcripts.

Can I use different ASR and TTS providers with this implementation?

Yes, the repository's provider-agnostic architecture supports multiple backends. The backend/utils/providers/ directory contains adapter patterns for OpenAI Whisper and Siliconflow SenseVoice (ASR), as well as OpenAI's TTS models. You configure providers in backend/config.js without modifying the core pipeline logic in backend/server.js.

How do control markers work in the controllable-TTS system?

Control markers are special text annotations like [EMO:happy] or [SPEED:fast] that the LLM embeds in its responses. The markup.py parser extracts these markers and maps them to voice parameters using voice_library.py before passing instructions to the TTS service, enabling dynamic adjustment of emotional tone and speech rate without requiring multiple model fine-tuning iterations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}