How the VAD to STT to LLM to TTS Pipeline Works in speech-to-speech

The VAD to STT to LLM to TTS pipeline is an asynchronous, queue-driven architecture that processes live audio through four stages—Voice Activity Detection, Speech-to-Text, Large Language Model, and Text-to-Speech—using threaded handlers that communicate via typed queues to enable real-time speech-to-speech conversation.

The huggingface/speech-to-speech repository implements a modular, real-time speech-to-speech system that converts microphone input into synthesized responses through a carefully orchestrated VAD to STT to LLM to TTS pipeline. This architecture enables low-latency conversational AI by streaming audio through four distinct processing stages connected by thread-safe queues.

Pipeline Architecture Overview

The pipeline consists of four specialized handlers that transform audio into speech:

• VAD (Voice Activity Detection) – Detects when a user starts and stops speaking, buffers audio fragments, and optionally publishes interim transcription events. Implemented in speech_to_speech/VAD/vad_handler.py.

• STT (Speech-to-Text) – Converts voiced audio chunks into text using multiple supported backends (Whisper, Faster-Whisper, Paraformer). See speech_to_speech/STT/whisper_stt_handler.py for the reference implementation.

• LLM (Large Language Model) – Receives transcriptions, optionally enriches them with chat history, and generates textual responses. Core logic resides in speech_to_speech/LLM/language_model.py, with an API-compatible variant in speech_to_speech/LLM/responses_api_language_model.py.

• TTS (Text-to-Speech) – Synthesizes LLM responses back into audio using models like Qwen-3, Pocket, or Kokoro. Example implementation: speech_to_speech/TTS/qwen3_tts_handler.py.

The orchestration layer in speech_to_speech/s2s_pipeline.py wires these components together using initialize_queues_and_events() and build_pipeline() functions.

Asynchronous Queue-Driven Orchestration

The pipeline uses a thread-based concurrency model managed by ThreadManager (speech_to_speech/utils/thread_manager.py). Each handler runs in its own thread and communicates through typed queue.Queue objects.

The data flow follows this path:

Audio chunks → VADHandler → VADOutItem (Queue)
               ↓
               STTHandler → STTOutItem (raw transcription)
               ↓
               TranscriptionNotifier (adds timestamps)
               ↓
               LLMHandler → LMOutItem (streaming response)
               ↓
               LMOutputProcessor → TTSInItem (post-processed text)
               ↓
               TTSHandler → AudioOutItem (synthesized speech)

This design provides natural back-pressure handling—if a downstream stage slows down, the upstream stage throttles automatically because the queue blocks on put() operations.

Stage-by-Stage Implementation

Voice Activity Detection

The VADHandler class processes raw audio chunks using the Silero VAD model via VADIterator (speech_to_speech/VAD/vad_iterator.py). It emits SpeechStartedEvent and SpeechStoppedEvent markers to signal turn boundaries. When enable_realtime_transcription is active, the handler publishes interim text events to text_output_queue while still listening.

Speech-to-Text Processing

STT handlers inherit from BaseSTTHandler (speech_to_speech/STT/base_stt_handler.py). The whisper_stt_handler.py implementation loads OpenAI Whisper models and processes the buffered audio returned by VAD. The system supports hot-swapping STT backends through get_stt_handler() factory functions mapped to CLI arguments like --stt whisper.

Language Model Inference

The LanguageModelHandler manages conversation state and streams tokens from transformer-based models (e.g., Qwen-3). The ResponsesAPILanguageModel variant (responses_api_language_model.py) wraps OpenAI's responses API for cloud-based inference. Both implementations yield LMOutItem objects containing partial responses.

Text-to-Speech Synthesis

TTS handlers receive processed text from LMOutputProcessor (speech_to_speech/LLM/lm_output_processor.py), which handles speculative turn detection and text compaction. The qwen3_tts_handler.py demonstrates how to convert streaming text into audio chunks using the Qwen-3 TTS model, outputting AudioOutItem objects for playback.

Advanced Pipeline Features

Speculative Turns – The SpeculativeTurnTracker (pipeline/speculative_turns.py) enables the system to treat partial LLM responses as tentative "turns," allowing early audio playback while the LLM continues generating. The VAD handler can merge speculative audio prefixes with final output to eliminate gaps.

Realtime Transcription Support – When enabled, the VAD stage emits interim transcription events via the text_output_queue, integrating with the OpenAI realtime server implementation (api/openai_realtime/server.py).

Graceful Shutdown – A CancelScope and signal handlers ensure all threads stop cleanly when the process receives SIGINT or SIGTERM, preventing audio device locks.

Running the Pipeline

To launch the complete pipeline from Python:

from speech_to_speech.s2s_pipeline import (
    parse_arguments,
    prepare_all_args,
    initialize_queues_and_events,
    build_pipeline,
)

# Parse CLI flags or construct ParsedArguments manually

args = parse_arguments()

# Initialize device-specific arguments for all handlers

prepare_all_args(
    args.module_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    # ... additional handler kwargs

)

# Create shared queues and threading events

queues = initialize_queues_and_events()

# Construct the handler chain (VAD → STT → LLM → TTS)

pipeline_manager = build_pipeline(
    args.module_kwargs,
    args.vad_handler_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    queues,
)

# Start and block until shutdown signal

pipeline_manager.start()
pipeline_manager.wait()

Running this with default arguments opens the microphone, processes speech through Silero VAD → Whisper STT → Qwen-3 LLM → Qwen-3 TTS, and plays the synthesized response.

To run the OpenAI realtime WebSocket server:

# Same initialization as above...

pipeline_manager = build_pipeline(
    args.module_kwargs,
    args.socket_receiver_kwargs,
    args.socket_sender_kwargs,
    args.websocket_streamer_kwargs,
    args.vad_handler_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    queues,
)

# Listens on ws://127.0.0.1:8765 using OpenAI realtime protocol

pipeline_manager.start()
pipeline_manager.wait()

Summary

• The VAD to STT to LLM to TTS pipeline uses threaded handlers connected by typed queues to process audio asynchronously.
• Back-pressure is handled naturally through blocking queue operations, preventing memory overflow during processing bottlenecks.
• Speculative turns enable low-latency responses by streaming partial LLM outputs to TTS before generation completes.
• The architecture supports pluggable backends for STT, LLM, and TTS via factory functions in s2s_pipeline.py.
• Graceful shutdown is managed through CancelScope and signal handling to ensure clean thread termination.

Frequently Asked Questions

How does the pipeline handle back-pressure between stages?

Each stage pulls from an input queue.Queue and pushes to an output queue using blocking put() operations. If the TTS handler slows down, the LLM handler blocks on its output queue, naturally throttling the entire upstream chain without explicit rate-limiting logic.

What is the purpose of speculative turns in the speech-to-speech pipeline?

The SpeculativeTurnTracker treats partial LLM responses as tentative conversation turns, allowing the TTS handler to begin synthesizing audio before the LLM finishes generating. This reduces perceived latency by overlapping LLM inference with audio playback, merging speculative prefixes with final outputs to avoid audible gaps.

Can I swap individual components like using a different STT model?

Yes. The build_pipeline() function uses factory methods (get_stt_handler(), get_llm_handler(), get_tts_handler()) mapped to CLI arguments (--stt, --llm_backend, --tts). Adding a new model requires implementing the BaseHandler interface and registering it in the handler mapping within s2s_pipeline.py.

How do I run the pipeline with the OpenAI realtime protocol?

Pass websocket_streamer_kwargs to build_pipeline() and ensure the LocalAudioStreamer is replaced with WebSocket handlers. The server implementation in speech_to_speech/api/openai_realtime/server.py exposes a WebSocket endpoint (default ws://127.0.0.1:8765) that accepts audio streams and returns synthesized speech using the standard OpenAI realtime API format.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}