# How to Set Up the Speech-to-Speech Pipeline: Complete Installation and Configuration Guide

> Learn how to set up the speech-to-speech pipeline with this complete installation and configuration guide. Get real-time voice translation working in minutes.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**TL;DR:** Install `pip install speech-to-speech`, set your `OPENAI_API_KEY`, and run `speech-to-speech serve` to start the realtime server, then `speech-to-speech talk` to connect a microphone client.

The **Speech-to-Speech** pipeline from Hugging Face is a modular, low-latency voice agent system that chains together voice activity detection, speech-to-text, language model inference, and text-to-synthesis in a multi-threaded architecture. This guide walks you through how to set up the Speech-to-Speech pipeline using CLI commands, custom backend configurations, and programmatic Python APIs.

## Quickstart: Minimal Setup

The fastest way to get running uses the defaults built into [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L69-L70】.

1. **Install the package**

```bash
pip install speech-to-speech

```

2. **Configure API access**

```bash
export OPENAI_API_KEY=sk-your-key-here

```

3. **Start the server**

```bash
speech-to-speech serve

```

4. **Connect a client** (in a new terminal)

```bash
speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime

```

The server exposes an OpenAI Realtime-compatible WebSocket at `ws://localhost:8765/v1/realtime` 【/cache/repos/github.com/huggingface/speech-to-speech/main/README.md†L26-L46】.

## Understanding the Pipeline Architecture

When you set up the Speech-to-Speech pipeline, you're configuring four stages that run in independent threads connected by queues:

| Stage | Purpose | Default Backend |
|-------|---------|---------------|
| **VAD** | Detect speech boundaries and turn-taking | Silero VAD v5 |
| **STT** | Transcribe user speech | Parakeet-TDT |
| **LLM** | Generate assistant responses | OpenAI Responses API |
| **TTS** | Synthesize output audio | Qwen3-TTS |

The `run_pipeline_command()` function in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) orchestrates this by parsing arguments through `HfArgumentParser` and calling either `build_pipeline()` or `build_local_pipeline()` 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L170-L190】.

### How Backend Selection Works

The [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) maps CLI flags to concrete handler implementations via `BackendSpec` objects. When you specify `--stt faster-whisper`, the registry instantiates the matching handler class.

The handler chain is constructed in `_build_handlers()` as: **VAD → STT → (optional TranscriptionNotifier) → LLM → LMOutputProcessor → TTS** 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L69-L110】【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L128-L155】.

## Deployment Modes

### Serve Mode: Production Backend

Use this when deploying the Speech-to-Speech pipeline as a standalone service:

```bash
speech-to-speech serve \
    --host 0.0.0.0 \
    --port 8765 \
    --stt parakeet-tdt \
    --llm_backend responses-api \
    --model_name gpt-4o-mini \
    --tts qwen3

```

The `ThreadManager` class handles graceful startup and shutdown of all handler threads 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L443-L470】.

### Talk Mode: Client Connection

Connect to a running server with the bundled microphone/speaker client:

```bash
speech-to-speech talk --url ws://your-server:8765/v1/realtime

```

### Local Mode: Single-Process Demo

Run server and client together for development:

```bash
speech-to-speech local

```

For Apple Silicon optimization, use the preset flag:

```bash
speech-to-speech local --mac-optimal-settings

```

## Custom Backend Configuration

### Using Local LLM with llama.cpp

Route the LLM stage to a local server while keeping cloud STT/TTS:

```bash
speech-to-speech serve \
    --stt faster-whisper \
    --llm_backend responses-api \
    --responses_api_base_url http://127.0.0.1:8000/v1 \
    --responses_api_api_key "" \
    --model_name ggml-org/gemma-4-E4B-it-GGUF \
    --tts pocket \
    --pocket_tts_voice jean

```

This configuration is parsed through the arguments classes in [`module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/module_arguments.py) 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/arguments_classes/module_arguments.py†L12-L48】.

### All Supported Backend Options

| Component | CLI Flag | Available Options |
|-----------|----------|-------------------|
| STT | `--stt` | `parakeet-tdt`, `whisper`, `faster-whisper`, `whisper-mlx`, `paraformer`, etc. |
| LLM | `--llm_backend` | `responses-api`, `transformers`, `mlx-lm`, `chat-completions` |
| TTS | `--tts` | `qwen3`, `kokoro-82m`, `pocket`, `chattts`, `mms-tts` |

The complete compatibility matrix is documented in the README 【/cache/repos/github.com/huggingface/speech-to-speech/main/README.md†L61-L78】.

## Programmatic Setup in Python

For integration into larger applications, call `run_pipeline_command()` directly:

```python
from speech_to_speech.s2s_pipeline import run_pipeline_command

# Equivalent to CLI: speech-to-speech serve --stt parakeet-tdt ...

run_pipeline_command(
    command="serve",
    argv=[
        "--stt", "faster-whisper",
        "--llm_backend", "mlx-lm",
        "--model_name", "mlx-community/Qwen3-8B-4bit",
        "--tts", "qwen3",
        "--log_level", "debug",
        "--speculative_reopen_ms", "150",
    ],
)

```

This bypasses the CLI entry point while using the same validation and pipeline construction logic.

## Advanced: Smart Turn and Latency Optimization

The Speech-to-Speech pipeline includes **Smart Turn** validation using an ONNX model to confirm Silero VAD decisions, reducing false turn transitions. Configure speculative processing with:

```bash
speech-to-speech serve \
    --smart_turn_enabled \
    --speculative_reopen_ms 200 \
    --smart_turn_threshold_confidence 0.75

```

The `SpeculativeTurnTracker` in [`pipeline/speculative_turns.py`](https://github.com/huggingface/speech-to-speech/blob/main/pipeline/speculative_turns.py) manages this logic.

## Offline Operation

After initial model download, run completely offline:

```bash

# First run downloads all models

speech-to-speech serve --stt parakeet-tdt --tts qwen3

# Subsequent runs work without internet

export HF_HUB_OFFLINE=1
speech-to-speech serve

```

## Summary

- **Install once**: `pip install speech-to-speech` provides all default dependencies
- **Three modes**: `serve` for backend deployment, `talk` for client connection, `local` for combined operation
- **Pluggable backends**: Select STT/LLM/TTS implementations via CLI flags mapped through [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)
- **Thread-safe architecture**: Four pipeline stages run independently with queue-based communication managed by `ThreadManager`
- **Production ready**: OpenAI Realtime-compatible WebSocket API with Smart Turn optimization

## Frequently Asked Questions

### What hardware requirements are needed to run the Speech-to-Speech pipeline?

Minimum requirements depend on your backend selection. The default configuration (Parakeet-TDT → OpenAI API → Qwen3-TTS) offloads LLM inference to OpenAI's servers, requiring only CPU for local audio processing. For fully local operation with `mlx-lm` or `transformers` backends, Apple Silicon with 16GB RAM or CUDA GPU with 8GB VRAM is recommended. The `--mac-optimal-settings` flag automatically configures MLX-accelerated backends for Apple hardware.

### How do I add custom voices or TTS models?

The TTS backend system in [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) supports voice selection through backend-specific arguments like `--pocket_tts_voice` or `--qwen3_voice_id`. To add a completely new TTS provider, implement the handler interface and register it in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py). The modular design allows swapping TTS without modifying other pipeline stages.

### Can the pipeline handle multiple simultaneous conversations?

The `serve` mode accepts multiple WebSocket connections, but each connection spawns an independent pipeline instance with its own thread pool. For high-concurrency deployments, run multiple server processes behind a load balancer. The `ThreadManager` provides clean per-connection cleanup when clients disconnect 【/cache/repos/github.com/huggingface/speech-to-speech/main/src/speech_to_speech/s2s_pipeline.py†L443-L470】.

### How do I debug pipeline issues or view component logs?

Set `--log_level debug` to enable verbose output from all handlers. Each pipeline stage (VAD, STT, LLM, TTS) logs queue depths, processing latency, and error conditions. For deeper inspection, use the programmatic API to inject custom logging into the handler chain in `_build_handlers()`.