# How to Configure and Swap Speech-to-Text (STT) Backends in the Speech-to-Speech Pipeline

> Easily configure and swap Speech-to-Text STT backends in the Hugging Face speech-to-speech pipeline. Learn how to set the stt field in ModuleArguments for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**You configure the STT backend by setting the `stt` field in `ModuleArguments` to one of seven supported values (`whisper`, `whisper-mlx`, `mlx-audio-whisper`, `paraformer`, `faster-whisper`, `parakeet-tdt`), and the `get_stt_handler` function in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) instantiates the appropriate handler class automatically.**

The **huggingface/speech-to-speech** repository provides a modular pipeline that lets you swap Speech-to-Text backends without rewriting core logic. Whether you need OpenAI Whisper for general accuracy, MLX-accelerated variants for Apple Silicon speed, or Parakeet TDT for low-latency streaming, backend selection happens through a single configuration point.

## Supported STT Backends and Their Handlers

The **STT backend selection logic** resides in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) inside the `get_stt_handler` function. This function branches on `module_kwargs.stt` and returns the corresponding concrete handler:

| `stt` value | Handler class | Best for |
|-------------|---------------|----------|
| `whisper` | `WhisperSTTHandler` ([`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py)) | General-purpose, CPU/GPU |
| `whisper-mlx` | `LightningWhisperSTTHandler` ([`lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/lightning_whisper_mlx_handler.py)) | macOS Apple Silicon acceleration |
| `mlx-audio-whisper` | `MLXAudioWhisperSTTHandler` ([`mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_audio_whisper_handler.py)) | Alternative MLX implementation |
| `paraformer` | `ParaformerSTTHandler` ([`paraformer_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/paraformer_handler.py)) | Chinese speech recognition |
| `faster-whisper` | `FasterWhisperSTTHandler` ([`faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/faster_whisper_handler.py)) | Optimized Inference with CTranslate2 |
| `parakeet-tdt` | `ParakeetTDTSTTHandler` ([`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py)) | Real-time streaming (default) |

Any unsupported value raises a `ValueError` with a descriptive message listing valid options.

All handlers implement the abstract interface defined in [`src/speech_to_speech/STT/base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/base_stt_handler.py), ensuring consistent input/output contracts through `STTIn` and `STTOut` types.

## Configuring STT Backends via CLI

The fastest way to **configure and swap STT backends** is through the built-in entry-point scripts. Use the `--stt` flag followed by backend-specific arguments:

### Whisper (Standard)

```bash
python -m speech_to_speech.scripts.listen_and_play \
    --stt whisper \
    --whisper_stt_handler_model large-v2 \
    --whisper_stt_handler_language en

```

### Whisper-MLX (Apple Silicon)

```bash
python -m speech_to_speech.scripts.listen_and_play \
    --stt whisper-mlx \
    --whisper_stt_handler_model tiny.en

```

### Parakeet TDT (Streaming Optimized)

```bash
python -m speech_to_speech.scripts.listen_and_play_realtime \
    --stt parakeet-tdt \
    --parakeet_tdt_stt_handler_model large

```

### Paraformer (Chinese)

```bash
python -m speech_to_speech.scripts.listen_and_play \
    --stt paraformer \
    --paraformer_stt_handler_model paraformer-zh

```

**Backend-specific argument discovery:** The CLI uses a pre-parse step to detect your chosen backend, then registers only the relevant handler arguments. Prefix flags with the handler name—e.g., `--whisper_stt_handler_device` or `--faster_whisper_stt_handler_compute_type`.

## Configuring STT Backends Programmatically

For **embedded pipeline usage**, construct the `ModuleArguments` dataclass and pass it to `get_stt_handler`:

```python
from threading import Event
from queue import Queue

from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.arguments_classes.whisper_stt_arguments import WhisperSTTHandlerArguments
from speech_to_speech.s2s_pipeline import get_stt_handler

# 1. Define top-level module configuration

module_cfg = ModuleArguments(
    stt="whisper",  # Swap to "paraformer", "parakeet-tdt", etc.

    enable_live_transcription=False,
)

# 2. Provide backend-specific configuration

whisper_cfg = WhisperSTTHandlerArguments(
    model="large-v2",
    language="en",
    device="cuda",
)

# 3. Instantiate the handler

stt_handler = get_stt_handler(
    module_kwargs=module_cfg,
    stop_event=Event(),
    spoken_prompt_queue=Queue(),
    text_prompt_queue=Queue(),
    speculative_turns=None,
    whisper_stt_handler_kwargs=whisper_cfg,
    faster_whisper_stt_handler_kwargs=None,
    paraformer_stt_handler_kwargs=None,
    mlx_audio_whisper_stt_handler_kwargs=None,
    parakeet_tdt_stt_handler_kwargs=None,
)

```

To **swap STT backends programmatically**, change only `module_cfg.stt` and supply the matching arguments dataclass. No pipeline rewiring required.

## Backend-Specific Configuration Reference

Each handler receives its settings through a `setup_kwargs` dictionary built from `vars(handler_kwargs)`. Key files for per-backend options:

- **WhisperSTTHandler**: [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py)
- **LightningWhisperSTTHandler**: [`src/speech_to_speech/arguments_classes/lightning_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/lightning_whisper_stt_arguments.py)
- **MLXAudioWhisperSTTHandler**: [`src/speech_to_speech/arguments_classes/mlx_audio_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/mlx_audio_whisper_stt_arguments.py)
- **ParaformerSTTHandler**: [`src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py)
- **FasterWhisperSTTHandler**: [`src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py)
- **ParakeetTDTSTTHandler**: [`src/speech_to_speech/arguments_classes/parakeet_tdt_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/parakeet_tdt_stt_arguments.py)

Note that `ParakeetTDTSTTHandler` additionally respects `module_kwargs.enable_live_transcription`, while `MLXAudioWhisperSTTHandler` inherits language settings from Whisper configuration.

## Runtime Backend Swapping (Advanced)

The pipeline initializes the STT handler once at startup. For **dynamic backend swapping** without process restart:

1. Signal shutdown with `stop_event.set()`
2. Drain the thread-safe queues (`spoken_prompt_queue`, `text_prompt_queue`)
3. Reconstruct the pipeline with new `ModuleArguments`
4. Restart with the new handler

Each handler runs in an isolated thread communicating through queues, making transitions safe when queues are properly flushed.

## Summary

- **Seven STT backends** are supported: `whisper`, `whisper-mlx`, `mlx-audio-whisper`, `paraformer`, `faster-whisper`, `parakeet-tdt`
- **Selection mechanism**: Set `ModuleArguments.stt` and let `get_stt_handler` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) instantiate the correct class
- **CLI method**: Use `--stt <backend>` with prefixed handler arguments
- **Programmatic method**: Build `ModuleArguments` and handler-specific dataclasses, then pass to `get_stt_handler`
- **Common interface**: All handlers extend `BaseSTTHandler` from [`base_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/base_stt_handler.py)

## Frequently Asked Questions

### What is the default STT backend if I don't specify one?

The default backend is `whisper` when no `--stt` flag is provided. However, the [`listen_and_play_realtime.py`](https://github.com/huggingface/speech-to-speech/blob/main/listen_and_play_realtime.py) script typically defaults to `parakeet-tdt` for optimal streaming latency. Check your specific entry-point's argument defaults in the source code, as defaults may evolve with releases.

### Can I use different STT backends for different languages?

Yes. The `paraformer` backend is optimized for Chinese speech recognition, while Whisper variants handle multilingual tasks well. Configure language-appropriate backends by setting `module_kwargs.stt` to `"paraformer"` for Chinese or `"whisper"` with the appropriate `--whisper_stt_handler_language` code for other languages.

### Why does my STT backend fail with a ValueError on startup?

A `ValueError` from `get_stt_handler` indicates an unrecognized `stt` value. Valid options are strictly: `whisper`, `whisper-mlx`, `mlx-audio-whisper`, `paraformer`, `faster-whisper`, and `parakeet-tdt`. Verify your spelling and ensure you're using the latest version of the repository, as new backends may be added.