How to Configure and Swap Speech-to-Text (STT) Backends in the Speech-to-Speech Pipeline

You configure the STT backend by setting the stt field in ModuleArguments to one of seven supported values (whisper, whisper-mlx, mlx-audio-whisper, paraformer, faster-whisper, parakeet-tdt), and the get_stt_handler function in src/speech_to_speech/s2s_pipeline.py instantiates the appropriate handler class automatically.

The huggingface/speech-to-speech repository provides a modular pipeline that lets you swap Speech-to-Text backends without rewriting core logic. Whether you need OpenAI Whisper for general accuracy, MLX-accelerated variants for Apple Silicon speed, or Parakeet TDT for low-latency streaming, backend selection happens through a single configuration point.

Supported STT Backends and Their Handlers

The STT backend selection logic resides in src/speech_to_speech/s2s_pipeline.py inside the get_stt_handler function. This function branches on module_kwargs.stt and returns the corresponding concrete handler:

stt value	Handler class	Best for
whisper	WhisperSTTHandler (whisper_stt_handler.py)	General-purpose, CPU/GPU
whisper-mlx	LightningWhisperSTTHandler (lightning_whisper_mlx_handler.py)	macOS Apple Silicon acceleration
mlx-audio-whisper	MLXAudioWhisperSTTHandler (mlx_audio_whisper_handler.py)	Alternative MLX implementation
paraformer	ParaformerSTTHandler (paraformer_handler.py)	Chinese speech recognition
faster-whisper	FasterWhisperSTTHandler (faster_whisper_handler.py)	Optimized Inference with CTranslate2
parakeet-tdt	ParakeetTDTSTTHandler (parakeet_tdt_handler.py)	Real-time streaming (default)

Any unsupported value raises a ValueError with a descriptive message listing valid options.

All handlers implement the abstract interface defined in src/speech_to_speech/STT/base_stt_handler.py, ensuring consistent input/output contracts through STTIn and STTOut types.

Configuring STT Backends via CLI

The fastest way to configure and swap STT backends is through the built-in entry-point scripts. Use the --stt flag followed by backend-specific arguments:

Whisper (Standard)

python -m speech_to_speech.scripts.listen_and_play \
    --stt whisper \
    --whisper_stt_handler_model large-v2 \
    --whisper_stt_handler_language en

Whisper-MLX (Apple Silicon)

python -m speech_to_speech.scripts.listen_and_play \
    --stt whisper-mlx \
    --whisper_stt_handler_model tiny.en

Parakeet TDT (Streaming Optimized)

python -m speech_to_speech.scripts.listen_and_play_realtime \
    --stt parakeet-tdt \
    --parakeet_tdt_stt_handler_model large

Paraformer (Chinese)

python -m speech_to_speech.scripts.listen_and_play \
    --stt paraformer \
    --paraformer_stt_handler_model paraformer-zh

Backend-specific argument discovery: The CLI uses a pre-parse step to detect your chosen backend, then registers only the relevant handler arguments. Prefix flags with the handler name—e.g., --whisper_stt_handler_device or --faster_whisper_stt_handler_compute_type.

Configuring STT Backends Programmatically

For embedded pipeline usage, construct the ModuleArguments dataclass and pass it to get_stt_handler:

from threading import Event
from queue import Queue

from speech_to_speech.arguments_classes.module_arguments import ModuleArguments
from speech_to_speech.arguments_classes.whisper_stt_arguments import WhisperSTTHandlerArguments
from speech_to_speech.s2s_pipeline import get_stt_handler

# 1. Define top-level module configuration

module_cfg = ModuleArguments(
    stt="whisper",  # Swap to "paraformer", "parakeet-tdt", etc.

    enable_live_transcription=False,
)

# 2. Provide backend-specific configuration

whisper_cfg = WhisperSTTHandlerArguments(
    model="large-v2",
    language="en",
    device="cuda",
)

# 3. Instantiate the handler

stt_handler = get_stt_handler(
    module_kwargs=module_cfg,
    stop_event=Event(),
    spoken_prompt_queue=Queue(),
    text_prompt_queue=Queue(),
    speculative_turns=None,
    whisper_stt_handler_kwargs=whisper_cfg,
    faster_whisper_stt_handler_kwargs=None,
    paraformer_stt_handler_kwargs=None,
    mlx_audio_whisper_stt_handler_kwargs=None,
    parakeet_tdt_stt_handler_kwargs=None,
)

To swap STT backends programmatically, change only module_cfg.stt and supply the matching arguments dataclass. No pipeline rewiring required.

Backend-Specific Configuration Reference

Each handler receives its settings through a setup_kwargs dictionary built from vars(handler_kwargs). Key files for per-backend options:

• WhisperSTTHandler: src/speech_to_speech/arguments_classes/whisper_stt_arguments.py
• LightningWhisperSTTHandler: src/speech_to_speech/arguments_classes/lightning_whisper_stt_arguments.py
• MLXAudioWhisperSTTHandler: src/speech_to_speech/arguments_classes/mlx_audio_whisper_stt_arguments.py
• ParaformerSTTHandler: src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py
• FasterWhisperSTTHandler: src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py
• ParakeetTDTSTTHandler: src/speech_to_speech/arguments_classes/parakeet_tdt_stt_arguments.py

Note that ParakeetTDTSTTHandler additionally respects module_kwargs.enable_live_transcription, while MLXAudioWhisperSTTHandler inherits language settings from Whisper configuration.

Runtime Backend Swapping (Advanced)

The pipeline initializes the STT handler once at startup. For dynamic backend swapping without process restart:

1. Signal shutdown with stop_event.set()
2. Drain the thread-safe queues (spoken_prompt_queue, text_prompt_queue)
3. Reconstruct the pipeline with new ModuleArguments
4. Restart with the new handler

Each handler runs in an isolated thread communicating through queues, making transitions safe when queues are properly flushed.

Summary

• Seven STT backends are supported: whisper, whisper-mlx, mlx-audio-whisper, paraformer, faster-whisper, parakeet-tdt
• Selection mechanism: Set ModuleArguments.stt and let get_stt_handler in s2s_pipeline.py instantiate the correct class
• CLI method: Use --stt <backend> with prefixed handler arguments
• Programmatic method: Build ModuleArguments and handler-specific dataclasses, then pass to get_stt_handler
• Common interface: All handlers extend BaseSTTHandler from base_stt_handler.py

Frequently Asked Questions

What is the default STT backend if I don't specify one?

The default backend is whisper when no --stt flag is provided. However, the listen_and_play_realtime.py script typically defaults to parakeet-tdt for optimal streaming latency. Check your specific entry-point's argument defaults in the source code, as defaults may evolve with releases.

Can I use different STT backends for different languages?

Yes. The paraformer backend is optimized for Chinese speech recognition, while Whisper variants handle multilingual tasks well. Configure language-appropriate backends by setting module_kwargs.stt to "paraformer" for Chinese or "whisper" with the appropriate --whisper_stt_handler_language code for other languages.

Why does my STT backend fail with a ValueError on startup?

A ValueError from get_stt_handler indicates an unrecognized stt value. Valid options are strictly: whisper, whisper-mlx, mlx-audio-whisper, paraformer, faster-whisper, and parakeet-tdt. Verify your spelling and ensure you're using the latest version of the repository, as new backends may be added.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}