# How the VAD to STT to LLM to TTS Pipeline Works in speech-to-speech

> Understand the VAD to STT to LLM to TTS pipeline for real-time speech-to-speech. Learn how this asynchronous architecture processes live audio through four stages for seamless conversation.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: architecture
- Published: 2026-08-07

---

**The VAD to STT to LLM to TTS pipeline is an asynchronous, queue-driven architecture that processes live audio through four stages—Voice Activity Detection, Speech-to-Text, Large Language Model, and Text-to-Speech—using threaded handlers that communicate via typed queues to enable real-time speech-to-speech conversation.**

The huggingface/speech-to-speech repository implements a modular, real-time speech-to-speech system that converts microphone input into synthesized responses through a carefully orchestrated VAD to STT to LLM to TTS pipeline. This architecture enables low-latency conversational AI by streaming audio through four distinct processing stages connected by thread-safe queues.

## Pipeline Architecture Overview

The pipeline consists of four specialized handlers that transform audio into speech:

- **VAD (Voice Activity Detection)** – Detects when a user starts and stops speaking, buffers audio fragments, and optionally publishes interim transcription events. Implemented in [`speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/VAD/vad_handler.py).

- **STT (Speech-to-Text)** – Converts voiced audio chunks into text using multiple supported backends (Whisper, Faster-Whisper, Paraformer). See [`speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/STT/whisper_stt_handler.py) for the reference implementation.

- **LLM (Large Language Model)** – Receives transcriptions, optionally enriches them with chat history, and generates textual responses. Core logic resides in [`speech_to_speech/LLM/language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/LLM/language_model.py), with an API-compatible variant in [`speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/LLM/responses_api_language_model.py).

- **TTS (Text-to-Speech)** – Synthesizes LLM responses back into audio using models like Qwen-3, Pocket, or Kokoro. Example implementation: [`speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/TTS/qwen3_tts_handler.py).

The orchestration layer in [`speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/s2s_pipeline.py) wires these components together using `initialize_queues_and_events()` and `build_pipeline()` functions.

## Asynchronous Queue-Driven Orchestration

The pipeline uses a **thread-based concurrency model** managed by `ThreadManager` ([`speech_to_speech/utils/thread_manager.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/utils/thread_manager.py)). Each handler runs in its own thread and communicates through typed `queue.Queue` objects.

The data flow follows this path:

```

Audio chunks → VADHandler → VADOutItem (Queue)
               ↓
               STTHandler → STTOutItem (raw transcription)
               ↓
               TranscriptionNotifier (adds timestamps)
               ↓
               LLMHandler → LMOutItem (streaming response)
               ↓
               LMOutputProcessor → TTSInItem (post-processed text)
               ↓
               TTSHandler → AudioOutItem (synthesized speech)

```

This design provides **natural back-pressure handling**—if a downstream stage slows down, the upstream stage throttles automatically because the queue blocks on `put()` operations.

## Stage-by-Stage Implementation

### Voice Activity Detection

The `VADHandler` class processes raw audio chunks using the Silero VAD model via `VADIterator` ([`speech_to_speech/VAD/vad_iterator.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/VAD/vad_iterator.py)). It emits `SpeechStartedEvent` and `SpeechStoppedEvent` markers to signal turn boundaries. When `enable_realtime_transcription` is active, the handler publishes interim text events to `text_output_queue` while still listening.

### Speech-to-Text Processing

STT handlers inherit from `BaseSTTHandler` ([`speech_to_speech/STT/base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/STT/base_stt_handler.py)). The [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py) implementation loads OpenAI Whisper models and processes the buffered audio returned by VAD. The system supports hot-swapping STT backends through `get_stt_handler()` factory functions mapped to CLI arguments like `--stt whisper`.

### Language Model Inference

The `LanguageModelHandler` manages conversation state and streams tokens from transformer-based models (e.g., Qwen-3). The `ResponsesAPILanguageModel` variant ([`responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/responses_api_language_model.py)) wraps OpenAI's responses API for cloud-based inference. Both implementations yield `LMOutItem` objects containing partial responses.

### Text-to-Speech Synthesis

TTS handlers receive processed text from `LMOutputProcessor` ([`speech_to_speech/LLM/lm_output_processor.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/LLM/lm_output_processor.py)), which handles speculative turn detection and text compaction. The [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) demonstrates how to convert streaming text into audio chunks using the Qwen-3 TTS model, outputting `AudioOutItem` objects for playback.

## Advanced Pipeline Features

**Speculative Turns** – The `SpeculativeTurnTracker` ([`pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/speculative_turns.py)) enables the system to treat partial LLM responses as tentative "turns," allowing early audio playback while the LLM continues generating. The VAD handler can merge speculative audio prefixes with final output to eliminate gaps.

**Realtime Transcription Support** – When enabled, the VAD stage emits interim transcription events via the `text_output_queue`, integrating with the OpenAI realtime server implementation ([`api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/server.py)).

**Graceful Shutdown** – A `CancelScope` and signal handlers ensure all threads stop cleanly when the process receives SIGINT or SIGTERM, preventing audio device locks.

## Running the Pipeline

To launch the complete pipeline from Python:

```python
from speech_to_speech.s2s_pipeline import (
    parse_arguments,
    prepare_all_args,
    initialize_queues_and_events,
    build_pipeline,
)

# Parse CLI flags or construct ParsedArguments manually

args = parse_arguments()

# Initialize device-specific arguments for all handlers

prepare_all_args(
    args.module_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    # ... additional handler kwargs

)

# Create shared queues and threading events

queues = initialize_queues_and_events()

# Construct the handler chain (VAD → STT → LLM → TTS)

pipeline_manager = build_pipeline(
    args.module_kwargs,
    args.vad_handler_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    queues,
)

# Start and block until shutdown signal

pipeline_manager.start()
pipeline_manager.wait()

```

Running this with default arguments opens the microphone, processes speech through Silero VAD → Whisper STT → Qwen-3 LLM → Qwen-3 TTS, and plays the synthesized response.

To run the OpenAI realtime WebSocket server:

```python

# Same initialization as above...

pipeline_manager = build_pipeline(
    args.module_kwargs,
    args.socket_receiver_kwargs,
    args.socket_sender_kwargs,
    args.websocket_streamer_kwargs,
    args.vad_handler_kwargs,
    args.whisper_stt_handler_kwargs,
    args.language_model_handler_kwargs,
    args.qwen3_tts_handler_kwargs,
    queues,
)

# Listens on ws://127.0.0.1:8765 using OpenAI realtime protocol

pipeline_manager.start()
pipeline_manager.wait()

```

## Summary

- The **VAD to STT to LLM to TTS pipeline** uses threaded handlers connected by typed queues to process audio asynchronously.
- **Back-pressure** is handled naturally through blocking queue operations, preventing memory overflow during processing bottlenecks.
- **Speculative turns** enable low-latency responses by streaming partial LLM outputs to TTS before generation completes.
- The architecture supports **pluggable backends** for STT, LLM, and TTS via factory functions in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py).
- **Graceful shutdown** is managed through `CancelScope` and signal handling to ensure clean thread termination.

## Frequently Asked Questions

### How does the pipeline handle back-pressure between stages?

Each stage pulls from an input `queue.Queue` and pushes to an output queue using blocking `put()` operations. If the TTS handler slows down, the LLM handler blocks on its output queue, naturally throttling the entire upstream chain without explicit rate-limiting logic.

### What is the purpose of speculative turns in the speech-to-speech pipeline?

The `SpeculativeTurnTracker` treats partial LLM responses as tentative conversation turns, allowing the TTS handler to begin synthesizing audio before the LLM finishes generating. This reduces perceived latency by overlapping LLM inference with audio playback, merging speculative prefixes with final outputs to avoid audible gaps.

### Can I swap individual components like using a different STT model?

Yes. The `build_pipeline()` function uses factory methods (`get_stt_handler()`, `get_llm_handler()`, `get_tts_handler()`) mapped to CLI arguments (`--stt`, `--llm_backend`, `--tts`). Adding a new model requires implementing the `BaseHandler` interface and registering it in the handler mapping within [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py).

### How do I run the pipeline with the OpenAI realtime protocol?

Pass `websocket_streamer_kwargs` to `build_pipeline()` and ensure the `LocalAudioStreamer` is replaced with WebSocket handlers. The server implementation in [`speech_to_speech/api/openai_realtime/server.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/api/openai_realtime/server.py) exposes a WebSocket endpoint (default `ws://127.0.0.1:8765`) that accepts audio streams and returns synthesized speech using the standard OpenAI realtime API format.