# How to Swap STT Backends Between Parakeet, Whisper, Faster Whisper, and Paraformer in Hugging Face Speech-to-Speech

> Easily swap STT backends in Hugging Face speech-to-speech between Parakeet, Whisper, Faster Whisper, and Paraformer by setting the stt field in ModuleArguments. Build your pipeline now.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**You can swap STT backends in the Hugging Face speech-to-speech library by setting the `stt` field in `ModuleArguments` to `parakeet-tdt`, `whisper`, `faster-whisper`, or `paraformer`, then passing the matching argument class to the pipeline builder.**

The **speech-to-speech** library provides a pluggable architecture for switching between multiple speech-to-text engines at runtime. This guide shows you exactly how to swap STT backends using either command-line flags or programmatic configuration, based on the actual implementation in the `huggingface/speech-to-speech` repository.

## How Backend Selection Works

The STT backend is determined at pipeline initialization. In [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 69-86 and 94-108), the function `get_stt_handler` inspects `ModuleArguments.stt` and instantiates the corresponding handler class:

| `stt` value | Handler class |
|-------------|-------------|
| `parakeet-tdt` | `ParakeetTDTSTTHandler` |
| `whisper` | `WhisperSTTHandler` |
| `whisper-mlx` | `LightningWhisperSTTHandler` |
| `mlx-audio-whisper` | `MLXAudioWhisperSTTHandler` |
| `faster-whisper` | `FasterWhisperSTTHandler` |
| `paraformer` | `ParaformerSTTHandler` |

Each backend has a dedicated argument dataclass (e.g., `WhisperSTTHandlerArguments`, `FasterWhisperSTTHandlerArguments`) that controls model selection, device placement, and generation options.

## Method 1: Swap STT Backends via Command Line

The simplest way to swap STT backends is through the CLI. The `--stt` flag maps directly to `ModuleArguments.stt`.

### Example: Switch to Whisper

```bash
python -m speech_to_speech.demo.server \
    --stt whisper \
    --whisper_model_name openai/whisper-base \
    --whisper_device cuda

```

### Example: Switch to Faster Whisper

```bash
python -m speech_to_speech.demo.server \
    --stt faster-whisper \
    --faster_whisper_stt_model_name openai/whisper-large-v3 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

```

### Example: Switch to Paraformer

```bash
python -m speech_to_speech.demo.server \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch

```

Backend-specific flags are automatically parsed into the appropriate argument dataclass. For instance, `--whisper_model_name` populates `WhisperSTTHandlerArguments.stt_model_name`.

## Method 2: Swap STT Backends Programmatically

For Python applications, import the argument classes and pass them to `build_pipeline()`. The key files are:

- [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) — defines `ModuleArguments.stt`
- [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py) — `WhisperSTTHandlerArguments`
- [`src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py) — `FasterWhisperSTTHandlerArguments`
- [`src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py) — `ParaformerSTTHandlerArguments`
- [`src/speech_to_speech/arguments_classes/parakeet_tdt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/parakeet_tdt_arguments.py) — `ParakeetTDTSTTHandlerArguments`

### Example: Configure Faster Whisper in Python

```python
from speech_to_speech.arguments_classes import (
    ModuleArguments,
    WhisperSTTHandlerArguments,
    FasterWhisperSTTHandlerArguments,
    ParaformerSTTHandlerArguments,
    ParakeetTDTSTTHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_pipeline

# 1. Select the backend

module_args = ModuleArguments(
    stt="faster-whisper",
)

# 2. Configure Faster Whisper options

faster_args = FasterWhisperSTTHandlerArguments(
    faster_whisper_stt_model_name="openai/whisper-large-v3",
    faster_whisper_stt_device="cuda",
    faster_whisper_stt_compute_type="float16",
)

# 3. Build pipeline (provide all argument objects; only the matching one is used)

pipeline = build_pipeline(
    module_kwargs=module_args,
    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),      # unused placeholder

    faster_whisper_stt_handler_kwargs=faster_args,                # active

    paraformer_stt_handler_kwargs=ParaformerSTTHandlerArguments(), # unused

    parakeet_tdt_stt_handler_kwargs=None,
)

```

Only the argument object matching `module_args.stt` is consulted; others are ignored but must be provided to satisfy the function signature.

### Example: Switch to Paraformer

```python
module_args = ModuleArguments(stt="paraformer")

paraformer_args = ParaformerSTTHandlerArguments(
    paraformer_stt_model_name="damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",
    paraformer_stt_device="cuda",
)

pipeline = build_pipeline(
    module_kwargs=module_args,
    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),
    faster_whisper_stt_handler_kwargs=FasterWhisperSTTHandlerArguments(),
    paraformer_stt_handler_kwargs=paraformer_args,  # active

    parakeet_tdt_stt_handler_kwargs=None,
)

```

## Backend-Specific Configuration Options

### Whisper (`whisper`)

- **Model**: Any Hugging Face Whisper model ID
- **Device**: `cpu`, `cuda`, `mps`
- **Key file**: [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py)

### Faster Whisper (`faster-whisper`)

- **Model**: Whisper model name or path
- **Device**: `cpu`, `cuda`
- **Compute type**: `int8`, `int8_float16`, `float16`, `float32`
- **Key file**: [`src/speech_to_speech/STT/faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/faster_whisper_handler.py)

### Paraformer (`paraformer`)

- **Model**: ModelScope model ID (e.g., `damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch`)
- **Device**: `cpu`, `cuda`
- **Key file**: [`src/speech_to_speech/STT/paraformer_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/paraformer_handler.py)

### Parakeet TDT (`parakeet-tdt`)

- **Model**: NVIDIA Parakeet TDT checkpoint
- **Device**: `cpu`, `cuda`
- **Key file**: [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py)

## Live Backend Switching

The pipeline reads `module_kwargs.stt` only at startup. To swap STT backends in a running server, you must:

1. Stop the current process
2. Relaunch with a different `--stt` value

There is no hot-swap mechanism in the current implementation. The `get_stt_handler` function executes once during `build_pipeline()` and returns the selected handler instance.

## Summary

- **Set `ModuleArguments.stt`** to `parakeet-tdt`, `whisper`, `faster-whisper`, or `paraformer` to select your backend
- **Match the argument class** to your chosen backend (e.g., `FasterWhisperSTTHandlerArguments` for `faster-whisper`)
- **Use CLI flags** for quick testing or **programmatic arguments** for production applications
- **Provide all argument objects** to `build_pipeline()` even if unused; only the matching backend receives its configuration
- **Restart required** to change backends; no runtime switching is supported

## Frequently Asked Questions

### Can I use multiple STT backends simultaneously in one pipeline?

No. The `get_stt_handler` function in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) selects exactly one STT handler based on `ModuleArguments.stt`. You cannot run multiple speech-to-text engines in parallel within a single pipeline instance.

### What happens if I pass the wrong argument class for my selected backend?

The pipeline ignores mismatched argument objects. If you set `stt="faster-whisper"` but only provide `WhisperSTTHandlerArguments`, the faster-whisper handler will use default values from `FasterWhisperSTTHandlerArguments()`. Always pair your `stt` value with the corresponding argument dataclass for predictable behavior.

### Is Whisper MLX different from regular Whisper?

Yes. `whisper-mlx` uses `LightningWhisperSTTHandler` from [`src/speech_to_speech/STT/lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/lightning_whisper_mlx_handler.py), which leverages Apple's MLX framework for optimized inference on Apple Silicon. It shares the same argument class as standard Whisper but routes to a different implementation optimized for Metal Performance Shaders.