How to Set Up the Speech-to-Speech Pipeline: Complete Installation and Configuration Guide

TL;DR: Install pip install speech-to-speech, set your OPENAI_API_KEY, and run speech-to-speech serve to start the realtime server, then speech-to-speech talk to connect a microphone client.

The Speech-to-Speech pipeline from Hugging Face is a modular, low-latency voice agent system that chains together voice activity detection, speech-to-text, language model inference, and text-to-synthesis in a multi-threaded architecture. This guide walks you through how to set up the Speech-to-Speech pipeline using CLI commands, custom backend configurations, and programmatic Python APIs.

Quickstart: Minimal Setup

The fastest way to get running uses the defaults built into s2s_pipeline.py 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L69-L70】.

1. Install the package

pip install speech-to-speech

2. Configure API access

export OPENAI_API_KEY=sk-your-key-here

3. Start the server

speech-to-speech serve

4. Connect a client (in a new terminal)

speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

The server exposes an OpenAI Realtime-compatible WebSocket at ws://localhost:8765/v1/realtime 【/cache/repos/github.com/huggingface/speech-to-speech/main/README.md†L26-L46】.

Understanding the Pipeline Architecture

When you set up the Speech-to-Speech pipeline, you're configuring four stages that run in independent threads connected by queues:

Stage	Purpose	Default Backend
VAD	Detect speech boundaries and turn-taking	Silero VAD v5
STT	Transcribe user speech	Parakeet-TDT
LLM	Generate assistant responses	OpenAI Responses API
TTS	Synthesize output audio	Qwen3-TTS

The run_pipeline_command() function in s2s_pipeline.py orchestrates this by parsing arguments through HfArgumentParser and calling either build_pipeline() or build_local_pipeline() 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L170-L190】.

How Backend Selection Works

The backend_registry.py maps CLI flags to concrete handler implementations via BackendSpec objects. When you specify --stt faster-whisper, the registry instantiates the matching handler class.

The handler chain is constructed in _build_handlers() as: VAD → STT → (optional TranscriptionNotifier) → LLM → LMOutputProcessor → TTS 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L69-L110】【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L128-L155】.

Deployment Modes

Serve Mode: Production Backend

Use this when deploying the Speech-to-Speech pipeline as a standalone service:

speech-to-speech serve \
    --host 0.0.0.0 \
    --port 8765 \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --model_name gpt-4o-mini \
    --tts qwen3

The ThreadManager class handles graceful startup and shutdown of all handler threads 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L443-L470】.

Talk Mode: Client Connection

Connect to a running server with the bundled microphone/speaker client:

speech-to-speech talk --url ws://your-server:8765/v1/realtime

Local Mode: Single-Process Demo

Run server and client together for development:

speech-to-speech local

For Apple Silicon optimization, use the preset flag:

speech-to-speech local --mac-optimal-settings

Custom Backend Configuration

Using Local LLM with llama.cpp

Route the LLM stage to a local server while keeping cloud STT/TTS:

speech-to-speech serve \
    --stt faster-whisper \
    --llm_backend responses-api \
    --responses_api_base_url http://127.0.0.1:8000/v1 \
    --responses_api_api_key "" \
    --model_name ggml-org/gemma-4-E4B-it-GGUF \
    --tts pocket \
    --pocket_tts_voice jean

This configuration is parsed through the arguments classes in module_arguments.py 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/arguments_classes/module_arguments.py†L12-L48】.

All Supported Backend Options

Component	CLI Flag	Available Options
STT	--stt	parakeet-tdt, whisper, faster-whisper, whisper-mlx, paraformer, etc.
LLM	--llm_backend	responses-api, transformers, mlx-lm, chat-completions
TTS	--tts	qwen3, kokoro-82m, pocket, chattts, mms-tts

The complete compatibility matrix is documented in the README 【/cache/repos/github.com/huggingface/speech-to-speech/main/README.md†L61-L78】.

Programmatic Setup in Python

For integration into larger applications, call run_pipeline_command() directly:

from speech_to_speech.s2s_pipeline import run_pipeline_command

# Equivalent to CLI: speech-to-speech serve --stt parakeet-tdt ...

run_pipeline_command(
    command="serve",
    argv=[
        "--stt", "faster-whisper",
        "--llm_backend", "mlx-lm",
        "--model_name", "mlx-community/Qwen3-8B-4bit",
        "--tts", "qwen3",
        "--log_level", "debug",
        "--speculative_reopen_ms", "150",
    ],
)

This bypasses the CLI entry point while using the same validation and pipeline construction logic.

Advanced: Smart Turn and Latency Optimization

The Speech-to-Speech pipeline includes Smart Turn validation using an ONNX model to confirm Silero VAD decisions, reducing false turn transitions. Configure speculative processing with:

speech-to-speech serve \
    --smart_turn_enabled \
    --speculative_reopen_ms 200 \
    --smart_turn_threshold_confidence 0.75

The SpeculativeTurnTracker in pipeline/speculative_turns.py manages this logic.

Offline Operation

After initial model download, run completely offline:

# First run downloads all models

speech-to-speech serve --stt parakeet-tdt --tts qwen3

# Subsequent runs work without internet

export HF_HUB_OFFLINE=1
speech-to-speech serve

Summary

• Install once: pip install speech-to-speech provides all default dependencies
• Three modes: serve for backend deployment, talk for client connection, local for combined operation
• Pluggable backends: Select STT/LLM/TTS implementations via CLI flags mapped through backend_registry.py
• Thread-safe architecture: Four pipeline stages run independently with queue-based communication managed by ThreadManager
• Production ready: OpenAI Realtime-compatible WebSocket API with Smart Turn optimization

Frequently Asked Questions

What hardware requirements are needed to run the Speech-to-Speech pipeline?

Minimum requirements depend on your backend selection. The default configuration (Parakeet-TDT → OpenAI API → Qwen3-TTS) offloads LLM inference to OpenAI's servers, requiring only CPU for local audio processing. For fully local operation with mlx-lm or transformers backends, Apple Silicon with 16GB RAM or CUDA GPU with 8GB VRAM is recommended. The --mac-optimal-settings flag automatically configures MLX-accelerated backends for Apple hardware.

How do I add custom voices or TTS models?

The TTS backend system in qwen3_tts_handler.py supports voice selection through backend-specific arguments like --pocket_tts_voice or --qwen3_voice_id. To add a completely new TTS provider, implement the handler interface and register it in backend_registry.py. The modular design allows swapping TTS without modifying other pipeline stages.

Can the pipeline handle multiple simultaneous conversations?

The serve mode accepts multiple WebSocket connections, but each connection spawns an independent pipeline instance with its own thread pool. For high-concurrency deployments, run multiple server processes behind a load balancer. The ThreadManager provides clean per-connection cleanup when clients disconnect 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L443-L470】.

How do I debug pipeline issues or view component logs?

Set --log_level debug to enable verbose output from all handlers. Each pipeline stage (VAD, STT, LLM, TTS) logs queue depths, processing latency, and error conditions. For deeper inspection, use the programmatic API to inject custom logging into the handler chain in _build_handlers().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}