How to Switch Between STT Backends in Hugging Face Speech-to-Speech: Parakeet, Whisper, Faster-Whisper, and Paraformer

Set the stt field in ModuleArguments to parakeet-tdt, whisper, faster-whisper, or paraformer, and provide the matching argument dataclass to instantiate your chosen backend.

The huggingface/speech-to-speech library uses a pluggable handler system that selects the speech-to-text engine at runtime. Understanding how to switch between STT backends lets you optimize for speed, accuracy, or hardware constraints without rewriting your pipeline code.

How Backend Selection Works

The core switching logic resides in src/speech_to_speech/s2s_pipeline.py (lines 69–86 and 94–108). When you call build_pipeline(), the helper function get_stt_handler inspects module_kwargs.stt and returns the appropriate handler class.

stt value	Handler class	Argument class
parakeet-tdt	ParakeetTDTSTTHandler	ParakeetTDTSTTHandlerArguments
whisper	WhisperSTTHandler	WhisperSTTHandlerArguments
whisper-mlx	LightningWhisperSTTHandler	WhisperSTTHandlerArguments
mlx-audio-whisper	MLXAudioWhisperSTTHandler	MLXAudioWhisperSTTHandlerArguments
faster-whisper	FasterWhisperSTTHandler	FasterWhisperSTTHandlerArguments
paraformer	ParaformerSTTHandler	ParaformerSTTHandlerArguments

Each handler imports from src/speech_to_speech/STT/ and receives configuration through a dedicated dataclass in src/speech_to_speech/arguments_classes/.

Method 1: Switch STT Backends via Command Line

The simplest way to switch STT backends is through the CLI. The --stt flag maps directly to ModuleArguments.stt, and backend-specific options are automatically routed to the correct argument class.

# Use OpenAI Whisper

python -m speech_to_speech.demo.server \
    --stt whisper \
    --whisper_model_name openai/whisper-base \
    --whisper_device cpu

# Use NVIDIA Parakeet-TDT (default)

python -m speech_to_speech.demo.server \
    --stt parakeet-tdt \
    --parakeet_tdt_stt_model_name nvidia/parakeet-tdt-1.1b

# Use Faster-Whisper for optimized inference

python -m speech_to_speech.demo.server \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v3 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

# Use Alibaba Paraformer

python -m speech_to_speech.demo.server \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer_asr_nat-zh-cn-16k-common-vocab8404-pytorch

Flag prefixes follow the pattern <backend>_stt_* and are defined in their respective *_stt_arguments.py files.

Method 2: Switch STT Backends Programmatically

For embedded applications or custom workflows, instantiate ModuleArguments and the matching handler arguments directly:

from speech_to_speech.arguments_classes import (
    ModuleArguments,
    WhisperSTTHandlerArguments,
    FasterWhisperSTTHandlerArguments,
    ParaformerSTTHandlerArguments,
    ParakeetTDTSTTHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_pipeline

# Example: Configure Faster-Whisper backend

module_args = ModuleArguments(stt="faster-whisper")

faster_whisper_args = FasterWhisperSTTHandlerArguments(
    faster_whisper_stt_model_name="openai/whisper-large-v3",
    faster_whisper_stt_device="cuda",
    faster_whisper_stt_compute_type="float16",
)

pipeline = build_pipeline(
    module_kwargs=module_args,
    # Unused handlers must still be passed (can use defaults)

    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),
    faster_whisper_stt_handler_kwargs=faster_whisper_args,
    paraformer_stt_handler_kwargs=ParaformerSTTHandlerArguments(),
    parakeet_tdt_stt_handler_kwargs=ParakeetTDTSTTHandlerArguments(),
    mlx_audio_whisper_stt_handler_kwargs=None,
    # LLM and TTS kwargs omitted for brevity

)

The pipeline only uses the argument object matching module_args.stt; others are ignored but required by the function signature.

Backend-Specific Configuration Files

Each STT backend has its own argument dataclass that defines available parameters:

• whisper_stt_arguments.py — stt_model_name, stt_device, stt_torch_dtype, stt_compile_mode
• faster_whisper_stt_arguments.py — faster_whisper_stt_model_name, faster_whisper_stt_device, faster_whisper_stt_compute_type, faster_whisper_stt_beam_size
• paraformer_stt_arguments.py — paraformer_stt_model_name, paraformer_stt_device, paraformer_stt_batch_size
• parakeet_tdt_arguments.py — parakeet_tdt_stt_model_name, parakeet_tdt_stt_device

Default values and validation are set in src/speech_to_speech/arguments_classes/module_arguments.py, which initializes stt: str = "parakeet-tdt".

Runtime Limitations and Hot-Swapping

The STT backend is locked at pipeline initialization. The get_stt_handler function reads module_kwargs.stt once during build_pipeline() and instantiates the corresponding class.

To change backends on a running server, you must:

1. Stop the current pipeline
2. Create new argument objects with a different stt value
3. Call build_pipeline() again

There is no internal mechanism for live backend switching without pipeline reconstruction.

Summary

• Switch STT backends by setting ModuleArguments.stt to parakeet-tdt, whisper, faster-whisper, paraformer, or MLX variants
• CLI usage: Pass --stt <backend> with prefixed flags like --whisper_model_name
• Programmatic usage: Instantiate the matching *STTHandlerArguments dataclass and pass to build_pipeline()
• Unused handler arguments are required in the signature but ignored at runtime
• Backend selection is resolved in s2s_pipeline.py via get_stt_handler at lines 69–86 and 94–108

Frequently Asked Questions

What is the default STT backend in speech-to-speech?

Parakeet-TDT is the default. In src/speech_to_speech/arguments_classes/module_arguments.py, the stt field defaults to "parakeet-tdt", which instantiates ParakeetTDTSTTHandler using NVIDIA's Parakeet model optimized for fast transcription.

Can I use multiple STT backends simultaneously in one pipeline?

No. The pipeline architecture supports only one active STT backend at a time. The get_stt_handler function returns a single handler instance based on module_kwargs.stt. To compare backends, run separate pipeline instances with different ModuleArguments configurations.

Why are unused handler arguments required in build_pipeline()?

The function signature of build_pipeline() in s2s_pipeline.py accepts all possible handler argument classes for type safety and forward compatibility. Arguments not matching the selected stt value are passed through but never instantiated into the active pipeline graph.

Which STT backend is fastest for CUDA GPUs?

Faster-Whisper typically offers the best throughput on NVIDIA GPUs due to optimized CTranslate2 kernels and configurable compute_type (INT8, FLOAT16). Parakeet-TDT also provides competitive speed with lower memory overhead. Benchmark your specific use case with --faster_whisper_stt_compute_type float16 versus parakeet-tdt to determine optimal latency for your hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}