# How to Switch Between STT Backends in Hugging Face Speech-to-Speech: Parakeet, Whisper, Faster-Whisper, and Paraformer

> Easily switch STT backends like Parakeet, Whisper, and Paraformer in Hugging Face Speech-to-Speech. Configure the stt field in ModuleArguments and set your preferred STT model.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-10

---

**Set the `stt` field in `ModuleArguments` to `parakeet-tdt`, `whisper`, `faster-whisper`, or `paraformer`, and provide the matching argument dataclass to instantiate your chosen backend.**

The **huggingface/speech-to-speech** library uses a pluggable handler system that selects the speech-to-text engine at runtime. Understanding how to switch between **STT backends** lets you optimize for speed, accuracy, or hardware constraints without rewriting your pipeline code.

## How Backend Selection Works

The core switching logic resides in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 69–86 and 94–108). When you call `build_pipeline()`, the helper function `get_stt_handler` inspects `module_kwargs.stt` and returns the appropriate handler class.

| `stt` value | Handler class | Argument class |
|-------------|-------------|--------------|
| `parakeet-tdt` | `ParakeetTDTSTTHandler` | `ParakeetTDTSTTHandlerArguments` |
| `whisper` | `WhisperSTTHandler` | `WhisperSTTHandlerArguments` |
| `whisper-mlx` | `LightningWhisperSTTHandler` | `WhisperSTTHandlerArguments` |
| `mlx-audio-whisper` | `MLXAudioWhisperSTTHandler` | `MLXAudioWhisperSTTHandlerArguments` |
| `faster-whisper` | `FasterWhisperSTTHandler` | `FasterWhisperSTTHandlerArguments` |
| `paraformer` | `ParaformerSTTHandler` | `ParaformerSTTHandlerArguments` |

Each handler imports from `src/speech_to_speech/STT/` and receives configuration through a dedicated dataclass in `src/speech_to_speech/arguments_classes/`.

## Method 1: Switch STT Backends via Command Line

The simplest way to **switch STT backends** is through the CLI. The `--stt` flag maps directly to `ModuleArguments.stt`, and backend-specific options are automatically routed to the correct argument class.

```bash

# Use OpenAI Whisper

python -m speech_to_speech.demo.server \
    --stt whisper \
    --whisper_model_name openai/whisper-base \
    --whisper_device cpu

# Use NVIDIA Parakeet-TDT (default)

python -m speech_to_speech.demo.server \
    --stt parakeet-tdt \
    --parakeet_tdt_stt_model_name nvidia/parakeet-tdt-1.1b

# Use Faster-Whisper for optimized inference

python -m speech_to_speech.demo.server \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v3 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

# Use Alibaba Paraformer

python -m speech_to_speech.demo.server \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer_asr_nat-zh-cn-16k-common-vocab8404-pytorch

```

Flag prefixes follow the pattern `<backend>_stt_*` and are defined in their respective `*_stt_arguments.py` files.

## Method 2: Switch STT Backends Programmatically

For embedded applications or custom workflows, instantiate `ModuleArguments` and the matching handler arguments directly:

```python
from speech_to_speech.arguments_classes import (
    ModuleArguments,
    WhisperSTTHandlerArguments,
    FasterWhisperSTTHandlerArguments,
    ParaformerSTTHandlerArguments,
    ParakeetTDTSTTHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_pipeline

# Example: Configure Faster-Whisper backend

module_args = ModuleArguments(stt="faster-whisper")

faster_whisper_args = FasterWhisperSTTHandlerArguments(
    faster_whisper_stt_model_name="openai/whisper-large-v3",
    faster_whisper_stt_device="cuda",
    faster_whisper_stt_compute_type="float16",
)

pipeline = build_pipeline(
    module_kwargs=module_args,
    # Unused handlers must still be passed (can use defaults)

    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),
    faster_whisper_stt_handler_kwargs=faster_whisper_args,
    paraformer_stt_handler_kwargs=ParaformerSTTHandlerArguments(),
    parakeet_tdt_stt_handler_kwargs=ParakeetTDTSTTHandlerArguments(),
    mlx_audio_whisper_stt_handler_kwargs=None,
    # LLM and TTS kwargs omitted for brevity

)

```

The pipeline **only uses the argument object matching `module_args.stt`**; others are ignored but required by the function signature.

## Backend-Specific Configuration Files

Each **STT backend** has its own argument dataclass that defines available parameters:

- **[`whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_arguments.py)** — `stt_model_name`, `stt_device`, `stt_torch_dtype`, `stt_compile_mode`
- **[`faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/faster_whisper_stt_arguments.py)** — `faster_whisper_stt_model_name`, `faster_whisper_stt_device`, `faster_whisper_stt_compute_type`, `faster_whisper_stt_beam_size`
- **[`paraformer_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/paraformer_stt_arguments.py)** — `paraformer_stt_model_name`, `paraformer_stt_device`, `paraformer_stt_batch_size`
- **[`parakeet_tdt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_arguments.py)** — `parakeet_tdt_stt_model_name`, `parakeet_tdt_stt_device`

Default values and validation are set in [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py), which initializes `stt: str = "parakeet-tdt"`.

## Runtime Limitations and Hot-Swapping

The **STT backend** is locked at pipeline initialization. The `get_stt_handler` function reads `module_kwargs.stt` once during `build_pipeline()` and instantiates the corresponding class. 

To **change backends on a running server**, you must:
1. Stop the current pipeline
2. Create new argument objects with a different `stt` value
3. Call `build_pipeline()` again

There is no internal mechanism for live backend switching without pipeline reconstruction.

## Summary

- **Switch STT backends** by setting `ModuleArguments.stt` to `parakeet-tdt`, `whisper`, `faster-whisper`, `paraformer`, or MLX variants
- **CLI usage**: Pass `--stt <backend>` with prefixed flags like `--whisper_model_name`
- **Programmatic usage**: Instantiate the matching `*STTHandlerArguments` dataclass and pass to `build_pipeline()`
- **Unused handler arguments** are required in the signature but ignored at runtime
- **Backend selection** is resolved in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) via `get_stt_handler` at lines 69–86 and 94–108

## Frequently Asked Questions

### What is the default STT backend in speech-to-speech?

**Parakeet-TDT** is the default. In [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py), the `stt` field defaults to `"parakeet-tdt"`, which instantiates `ParakeetTDTSTTHandler` using NVIDIA's Parakeet model optimized for fast transcription.

### Can I use multiple STT backends simultaneously in one pipeline?

No. The pipeline architecture supports **only one active STT backend** at a time. The `get_stt_handler` function returns a single handler instance based on `module_kwargs.stt`. To compare backends, run separate pipeline instances with different `ModuleArguments` configurations.

### Why are unused handler arguments required in build_pipeline()?

The function signature of `build_pipeline()` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) accepts all possible handler argument classes for type safety and forward compatibility. Arguments not matching the selected `stt` value are passed through but never instantiated into the active pipeline graph.

### Which STT backend is fastest for CUDA GPUs?

**Faster-Whisper** typically offers the best throughput on NVIDIA GPUs due to optimized CTranslate2 kernels and configurable `compute_type` (INT8, FLOAT16). **Parakeet-TDT** also provides competitive speed with lower memory overhead. Benchmark your specific use case with `--faster_whisper_stt_compute_type float16` versus `parakeet-tdt` to determine optimal latency for your hardware.