# How to Select Different STT Backends for Speech-to-Speech: A Complete Guide

> Effortlessly select STT backends for speech-to-speech using the --stt flag or programmatically. Explore options like whisper, faster-whisper, and mlx-audio-whisper for your Hugging Face pipeline.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Use the `--stt` flag with any registered backend name (e.g., `whisper`, `faster-whisper`, `mlx-audio-whisper`) or construct a `BackendSelection` object programmatically to switch speech-to-text engines in the Hugging Face Speech-to-Speech pipeline.**

The Hugging Face Speech-to-Speech (S2S) framework treats every pipeline component—including speech-to-text—as a **pluggable backend**. Whether you need faster inference, GPU acceleration on Apple Silicon, or specialized models for non-English languages, switching STT backends requires no code changes beyond a command-line flag or constructor argument.

## Where STT Backends Are Registered

All supported STT backends are centralized in **[`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py)**. The `STT_BACKENDS` dictionary maps short names to `BackendInfo` objects containing the module path and configuration prefix:

```python
STT_BACKENDS = {
    "whisper": BackendInfo(
        module="speech_to_speech.STT.whisper_stt_handler",
        config_prefix="stt",
        kind="stt",
    ),
    "whisper-mlx": BackendInfo(
        module="speech_to_speech.STT.lightning_whisper_mlx_handler",
        config_prefix="stt",
        kind="stt",
    ),
    "mlx-audio-whisper": BackendInfo(
        module="speech_to_speech.STT.mlx_audio_whisper_handler",
        config_prefix="mlx_audio_whisper_stt",
        kind="stt",
    ),
    "faster-whisper": BackendInfo(
        module="speech_to_speech.STT.faster_whisper_handler",
        config_prefix="faster_whisper_stt",
        kind="stt",
    ),
    "parakeet-tdt": BackendInfo(
        module="speech_to_speech.STT.parakeet_tdt_handler",
        config_prefix="parakeet_tdt",
        kind="stt",
    ),
    "paraformer": BackendInfo(
        module="speech_to_speech.STT.paraformer_handler",
        config_prefix="paraformer_stt",
        kind="stt",
    ),
}

```

Each entry specifies:
- **`module`**: Python import path to the handler class
- **`config_prefix`**: Namespace for CLI arguments specific to that backend
- **`kind`**: Backend category (`"stt"` for speech-to-text)

## How Runtime Selection Works

The selection mechanism spans two core files: [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) for definitions and **[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)** for orchestration.

### Command-Line Parsing

In [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), the argument parser constrains `--stt` to valid backend keys:

```python
_pre.add_argument(
    "--stt",
    choices=tuple(STT_BACKENDS),
    help="Select the STT backend (default is the module default)",
)

```

### Pipeline Instantiation

After parsing, the selected name becomes a `BackendSelection` object:

```python
stt_backend = BackendSelection(kind="stt", name=_stt_name)

```

This object feeds into `create_backend_handler(stt_backend, stt_context)`, which imports and instantiates the concrete handler (e.g., `WhisperSTTHandler` from [`STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/whisper_stt_handler.py)). The handler produces `STTOutItem` objects placed on `stt_output_queue`. Depending on `spec.capabilities.bypasses_transcription_notifier`, the pipeline either routes through a `TranscriptionNotifier` or sends raw text directly to the LLM queue.

## Selecting STT Backends from the Command Line

Pass the backend name to `--stt`, followed by backend-specific arguments prefixed according to the registry entry:

```bash

# Whisper (default, CPU-friendly)

speech-to-speech serve --stt whisper --language en

# MLX-Audio Whisper: GPU-accelerated on Apple Silicon

speech-to-speech serve --stt mlx-audio-whisper --device mps

# Faster-Whisper with German language model

speech-to-speech serve \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v3 \
    --faster_whisper_stt_gen_language de

# Parakeet TDT with automatic device selection

speech-to-speech serve \
    --stt parakeet-tdt \
    --parakeet_tdt_device auto

# Paraformer for streaming Mandarin or other CJK languages

speech-to-speech serve \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch

```

Backend-specific flags live in **`src/speech_to_speech/arguments_classes/`** (e.g., [`faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/faster_whisper_stt_arguments.py), [`paraformer_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/paraformer_stt_arguments.py)).

## Selecting STT Backends Programmatically

For embedded applications or custom orchestration, construct `BackendSelection` directly:

```python
from speech_to_speech.backend_registry import BackendSelection, STT_BACKENDS
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

# Select the "paraformer" backend

stt_backend = BackendSelection(kind="stt", name="paraformer")

pipeline = SpeechToSpeechPipeline(
    stt_backend=stt_backend,
    # ... additional required backends: llm_backend, tts_backend, etc.

)

pipeline.run()

```

Inspect the selection for debugging or dynamic configuration:

```python
print(stt_backend.spec.module)          # 'speech_to_speech.STT.paraformer_handler'

print(stt_backend.spec.config_prefix)   # 'paraformer_stt'

```

## Adding a Custom STT Backend

Extend the framework with proprietary or experimental models in three steps:

### Step 1: Implement the Handler

Subclass `BaseSTTHandler` in a new file under `src/speech_to_speech/STT/`:

```python

# src/speech_to_speech/STT/my_custom_handler.py

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler

class MyCustomSTTHandler(BaseSTTHandler):
    def __init__(self, args):
        super().__init__(args)
        # Initialize your ASR model here

        self.model = load_model(args.my_custom_stt_model_path)

    async def process(self, audio_chunk: bytes) -> str:
        """Transcribe audio chunk to text."""
        text = self.model.transcribe(audio_chunk)
        return text

```

### Step 2: Register the Backend

Add an entry to `STT_BACKENDS` in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py):

```python
STT_BACKENDS["my-custom"] = BackendInfo(
    module="speech_to_speech.STT.my_custom_handler",
    config_prefix="my_custom_stt",
    kind="stt",
)

```

### Step 3: (Optional) Define CLI Arguments

Create [`src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py):

```python
from dataclasses import dataclass

@dataclass
class MyCustomSTTArguments:
    my_custom_stt_model_path: str = "models/my-model"
    my_custom_stt_language: str = "auto"

```

The new backend is immediately available via `--stt my-custom`.

## Available STT Backends Reference

| Backend | Best For | Key File |
|---------|----------|----------|
| `whisper` | General-purpose, CPU inference | [`STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/whisper_stt_handler.py) |
| `whisper-mlx` | Apple Silicon, Lightning-fast inference | [`STT/lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/lightning_whisper_mlx_handler.py) |
| `mlx-audio-whisper` | Apple Silicon, unified audio toolkit | [`STT/mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/mlx_audio_whisper_handler.py) |
| `faster-whisper` | Production throughput, CTranslate2 | [`STT/faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/faster_whisper_handler.py) |
| `parakeet-tdt` | NVIDIA GPU-optimized streaming | [`STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/parakeet_tdt_handler.py) |
| `paraformer` | Streaming Mandarin/CJK recognition | [`STT/paraformer_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/paraformer_handler.py) |

## Summary

- **Registry location**: All STT backends are defined in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) as `STT_BACKENDS` entries
- **CLI selection**: Use `--stt <backend_name>` with backend-specific prefixed arguments
- **Programmatic selection**: Pass `BackendSelection(kind="stt", name="...")` to `SpeechToSpeechPipeline`
- **Extensibility**: Implement `BaseSTTHandler`, register in `STT_BACKENDS`, optionally add argument classes

## Frequently Asked Questions

### How do I find which STT backends are currently installed?

Check the `STT_BACKENDS` dictionary keys in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) or run `speech-to-speech serve --help` to see the `--stt` choices. The repository does not support runtime enumeration of handler modules; the registry hardcodes available options.

### Can I use multiple STT backends simultaneously in one pipeline?

No. The `SpeechToSpeechPipeline` accepts a single `stt_backend` argument. To A/B test backends, instantiate separate pipeline objects or implement a custom handler that internally multiplexes between multiple ASR models based on audio characteristics.

### What happens if I specify an STT backend without its required dependencies?

The handler import fails during `create_backend_handler()` with a `ModuleNotFoundError` or `ImportError`. Install backend-specific requirements (e.g., `pip install faster-whisper` for the `faster-whisper` backend) before selection. Dependencies are not enforced at the registry level.