# How to Swap Components in the Speech-to-Speech Pipeline: Complete Backend Customization Guide

> Customize the Hugging Face Speech-to-Speech pipeline by swapping components using command-line flags or JSON config. No code changes needed for backend customization.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Swapping components in the Hugging Face Speech-to-Speech pipeline requires only command-line flags or JSON configuration—no code changes needed—thanks to a modular backend registry system.**

The Speech-to-Speech repository (`huggingface/speech-to-speech`) implements a **pluggable pipeline architecture** where each major stage (STT, LLM, TTS) is a swappable backend. Understanding how to swap these components lets you optimize for latency, quality, or hardware constraints without touching the core pipeline logic.

## Understanding the Pipeline Architecture

The system processes audio through a fixed handler chain:

```

VAD → STT → (optional) TranscriptionNotifier → LLM → LMOutputProcessor → TTS

```

Each swappable stage conforms to a common interface and is registered in [`speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/speech_to_speech/backend_registry.py). The registries `STT_BACKENDS`, `LLM_BACKENDS`, and `TTS_BACKENDS` map **string names to factory functions**, enabling runtime selection.

## How Backend Selection Works

The pipeline uses a three-step resolution process:

1. **Argument parsing** — `speech_to_speech/s2s_pipeline.py::parse_arguments()` (lines 70-104) reads `--stt`, `--llm_backend`, and `--tts` flags
2. **Registry lookup** — `backend_registry.py::select_backend()` (lines 62-68) converts names to `BackendSelection` objects with normalized configuration via `BackendSpec.normalize`
3. **Handler instantiation** — `backend_registry.py::create_backend_handler()` (lines 82-88) executes the factory to build the concrete handler
4. **Chain assembly** — `s2s_pipeline.py::_build_handlers()` (lines 48-69) inserts handlers into the runtime queue structure

Because the registries use factory functions, adding or swapping backends requires zero changes to [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py).

## Swapping Components via Command Line

The simplest method uses flags when starting the server:

```bash

# Replace STT engine

speech-to-speech serve --stt whisper

# Use different language model

speech-to-speech serve --llm_backend mlx-lm

# Change TTS engine

speech-to-speech serve --tts qwen3

```

Combine multiple swaps in one command:

```bash
speech-to-speech serve \
  --stt paraformer \
  --llm_backend responses-api \
  --tts pocket \
  --log-level debug

```

## Swapping Components via JSON Configuration

For reproducible deployments, use a JSON configuration file:

```json
{
  "stt": "faster-whisper",
  "llm_backend": "chat-completions",
  "tts": "chatTTS",
  "stt_device": "cuda",
  "llm_device": "cuda",
  "tts_device": "cuda",
  "log_level": "info"
}

```

Launch with:

```bash
speech-to-speech serve pipeline.json

```

The `parse_arguments()` function detects JSON paths through its `_is_json` branch and merges settings with platform defaults.

## Swapping Components Programmatically

For dynamic configuration in Python applications:

```python
from speech_to_speech.s2s_pipeline import parse_arguments, run_pipeline_command

# Configure paraformer STT and kokoro TTS

cli_args = ["--stt", "parakeet-tdt", "--tts", "kokoro"]
parsed = parse_arguments(cli_args, command="serve")

# Verify selections

print(parsed.stt_backend.name)   # "parakeet-tdt"

print(parsed.llm_backend.name)   # default value

print(parsed.tts_backend.name)   # "kokoro"

# Execute pipeline

run_pipeline_command("serve", cli_args)

```

This approach enables runtime backend selection based on user preferences or hardware detection.

## Available Backend Options

The registry supports these backend categories (see [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) for current registrations):

**STT Backends:**
- `whisper` — OpenAI Whisper
- `whisper-mlx` — Apple Silicon optimized
- `faster-whisper` — Faster Whisper implementation
- `paraformer` — Alibaba Paraformer
- `parakeet-tdt` — NVIDIA Parakeet

**LLM Backends:**
- `mlx-lm` — Apple MLX framework
- `transformers` — Hugging Face Transformers
- `chat-completions` — OpenAI-compatible API
- `responses-api` — Responses API interface

**TTS Backends:**
- `qwen3` — Qwen3 TTS
- `chatTTS` — ChatTTS model
- `facebookMMS` — Meta MMS
- `pocket` — PocketSynthesis
- `kokoro` — Kokoro TTS

## Configuration File with Backend-Specific Options

Some backends require additional parameters:

```json
{
  "stt": "facebookMMS",
  "llm_backend": "transformers",
  "tts": "facebookMMS",
  "facebook_mms_tts_language": "en",
  "stt_device": "cpu",
  "llm_device": "cuda:0",
  "tts_device": "cpu",
  "log_level": "debug"
}

```

Backend-specific arguments are defined in `speech_to_speech/arguments_classes/` (e.g., [`whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_arguments.py)).

## Complete Working Examples

**Full CLI swap with debugging:**

```bash
speech-to-speech serve \
  --stt whisper-mlx \
  --llm_backend mlx-lm \
  --tts qwen3 \
  --log-level debug

```

**JSON configuration for multilingual deployment:**

```bash
cat > pipeline.json <<'EOF'
{
  "stt": "paraformer",
  "llm_backend": "transformers",
  "tts": "facebookMMS",
  "facebook_mms_tts_language": "en",
  "log_level": "info"
}
EOF
speech-to-speech serve pipeline.json

```

**Python API for conditional backend selection:**

```python
from speech_to_speech.s2s_pipeline import parse_arguments, run_pipeline_command
import torch

# Auto-select based on availability

stt_choice = "whisper-mlx" if torch.backends.mps.is_available() else "faster-whisper"
cli_args = ["--stt", stt_choice, "--llm_backend", "mlx-lm"]
run_pipeline_command("serve", cli_args)

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Pipeline orchestration, argument parsing, handler chain construction |
| [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) | Backend declarations, factory registration, configuration normalization |
| `src/speech_to_speech/arguments_classes/` | Typed dataclasses for backend-specific options |
| `src/speech_to_speech/STT/*.py` | Concrete STT implementations (e.g., [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py)) |
| `src/speech_to_speech/LLM/*.py` | Concrete LLM implementations (e.g., [`language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/language_model.py)) |
| `src/speech_to_speech/TTS/*.py` | Concrete TTS implementations (e.g., [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py)) |

## Summary

- **Command-line flags** (`--stt`, `--llm_backend`, `--tts`) provide instant backend swapping without code changes
- **JSON configuration files** enable reproducible, version-controlled pipeline deployments
- **Python API** (`parse_arguments`, `run_pipeline_command`) supports dynamic, conditional backend selection
- **Registry-based architecture** in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) decouples backend implementations from pipeline orchestration
- **Factory pattern** allows new backends to be added by registering name-to-constructor mappings

## Frequently Asked Questions

### What backends are available in the Speech-to-Speech pipeline?

According to the [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) source code, available backends include **STT options** (whisper, faster-whisper, paraformer, parakeet-tdt), **LLM options** (mlx-lm, transformers, chat-completions, responses-api), and **TTS options** (qwen3, chatTTS, facebookMMS, pocket, kokoro). Run `speech-to-speech serve --help` to see current registrations for your installation.

### Can I use different hardware devices for each pipeline stage?

Yes. The JSON configuration format accepts `stt_device`, `llm_device`, and `tts_device` fields to assign specific GPUs or CPU for each component. The `parse_arguments()` function passes these values through `BackendSpec.normalize` to handler constructors.

### How do I add a custom backend to the pipeline?

Create a handler class following the existing patterns in `src/speech_to_speech/STT/`, `LLM/`, or `TTS/`, then register it in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) by adding an entry to the appropriate `*_BACKENDS` dictionary with a factory function. No changes to [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) are required.

### Why does my JSON configuration merge with defaults instead of replacing them?

The `parse_arguments()` function in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) applies platform presets (notably macOS optimizations) before merging your JSON values. To override completely, explicitly set all relevant fields in your configuration file rather than relying on selective specification.