How to Swap STT Backends Between Parakeet, Whisper, Faster Whisper, and Paraformer in Hugging Face Speech-to-Speech

You can swap STT backends in the Hugging Face speech-to-speech library by setting the stt field in ModuleArguments to parakeet-tdt, whisper, faster-whisper, or paraformer, then passing the matching argument class to the pipeline builder.

The speech-to-speech library provides a pluggable architecture for switching between multiple speech-to-text engines at runtime. This guide shows you exactly how to swap STT backends using either command-line flags or programmatic configuration, based on the actual implementation in the huggingface/speech-to-speech repository.

How Backend Selection Works

The STT backend is determined at pipeline initialization. In src/speech_to_speech/s2s_pipeline.py (lines 69-86 and 94-108), the function get_stt_handler inspects ModuleArguments.stt and instantiates the corresponding handler class:

stt value	Handler class
parakeet-tdt	ParakeetTDTSTTHandler
whisper	WhisperSTTHandler
whisper-mlx	LightningWhisperSTTHandler
mlx-audio-whisper	MLXAudioWhisperSTTHandler
faster-whisper	FasterWhisperSTTHandler
paraformer	ParaformerSTTHandler

Each backend has a dedicated argument dataclass (e.g., WhisperSTTHandlerArguments, FasterWhisperSTTHandlerArguments) that controls model selection, device placement, and generation options.

Method 1: Swap STT Backends via Command Line

The simplest way to swap STT backends is through the CLI. The --stt flag maps directly to ModuleArguments.stt.

Example: Switch to Whisper

python -m speech_to_speech.demo.server \
    --stt whisper \
    --whisper_model_name openai/whisper-base \
    --whisper_device cuda

Example: Switch to Faster Whisper

python -m speech_to_speech.demo.server \
    --stt faster-whisper \
    --faster_whisper_stt_model_name openai/whisper-large-v3 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

Example: Switch to Paraformer

python -m speech_to_speech.demo.server \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch

Backend-specific flags are automatically parsed into the appropriate argument dataclass. For instance, --whisper_model_name populates WhisperSTTHandlerArguments.stt_model_name.

Method 2: Swap STT Backends Programmatically

For Python applications, import the argument classes and pass them to build_pipeline(). The key files are:

• src/speech_to_speech/arguments_classes/module_arguments.py — defines ModuleArguments.stt
• src/speech_to_speech/arguments_classes/whisper_stt_arguments.py — WhisperSTTHandlerArguments
• src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py — FasterWhisperSTTHandlerArguments
• src/speech_to_speech/arguments_classes/paraformer_stt_arguments.py — ParaformerSTTHandlerArguments
• src/speech_to_speech/arguments_classes/parakeet_tdt_arguments.py — ParakeetTDTSTTHandlerArguments

Example: Configure Faster Whisper in Python

from speech_to_speech.arguments_classes import (
    ModuleArguments,
    WhisperSTTHandlerArguments,
    FasterWhisperSTTHandlerArguments,
    ParaformerSTTHandlerArguments,
    ParakeetTDTSTTHandlerArguments,
)
from speech_to_speech.s2s_pipeline import build_pipeline

# 1. Select the backend

module_args = ModuleArguments(
    stt="faster-whisper",
)

# 2. Configure Faster Whisper options

faster_args = FasterWhisperSTTHandlerArguments(
    faster_whisper_stt_model_name="openai/whisper-large-v3",
    faster_whisper_stt_device="cuda",
    faster_whisper_stt_compute_type="float16",
)

# 3. Build pipeline (provide all argument objects; only the matching one is used)

pipeline = build_pipeline(
    module_kwargs=module_args,
    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),      # unused placeholder

    faster_whisper_stt_handler_kwargs=faster_args,                # active

    paraformer_stt_handler_kwargs=ParaformerSTTHandlerArguments(), # unused

    parakeet_tdt_stt_handler_kwargs=None,
)

Only the argument object matching module_args.stt is consulted; others are ignored but must be provided to satisfy the function signature.

Example: Switch to Paraformer

module_args = ModuleArguments(stt="paraformer")

paraformer_args = ParaformerSTTHandlerArguments(
    paraformer_stt_model_name="damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch",
    paraformer_stt_device="cuda",
)

pipeline = build_pipeline(
    module_kwargs=module_args,
    whisper_stt_handler_kwargs=WhisperSTTHandlerArguments(),
    faster_whisper_stt_handler_kwargs=FasterWhisperSTTHandlerArguments(),
    paraformer_stt_handler_kwargs=paraformer_args,  # active

    parakeet_tdt_stt_handler_kwargs=None,
)

Backend-Specific Configuration Options

Whisper (whisper)

• Model: Any Hugging Face Whisper model ID
• Device: cpu, cuda, mps
• Key file: src/speech_to_speech/STT/whisper_stt_handler.py

Faster Whisper (faster-whisper)

• Model: Whisper model name or path
• Device: cpu, cuda
• Compute type: int8, int8_float16, float16, float32
• Key file: src/speech_to_speech/STT/faster_whisper_handler.py

Paraformer (paraformer)

• Model: ModelScope model ID (e.g., damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch)
• Device: cpu, cuda
• Key file: src/speech_to_speech/STT/paraformer_handler.py

Parakeet TDT (parakeet-tdt)

• Model: NVIDIA Parakeet TDT checkpoint
• Device: cpu, cuda
• Key file: src/speech_to_speech/STT/parakeet_tdt_handler.py

Live Backend Switching

The pipeline reads module_kwargs.stt only at startup. To swap STT backends in a running server, you must:

1. Stop the current process
2. Relaunch with a different --stt value

There is no hot-swap mechanism in the current implementation. The get_stt_handler function executes once during build_pipeline() and returns the selected handler instance.

Summary

• Set ModuleArguments.stt to parakeet-tdt, whisper, faster-whisper, or paraformer to select your backend
• Match the argument class to your chosen backend (e.g., FasterWhisperSTTHandlerArguments for faster-whisper)
• Use CLI flags for quick testing or programmatic arguments for production applications
• Provide all argument objects to build_pipeline() even if unused; only the matching backend receives its configuration
• Restart required to change backends; no runtime switching is supported

Frequently Asked Questions

Can I use multiple STT backends simultaneously in one pipeline?

No. The get_stt_handler function in src/speech_to_speech/s2s_pipeline.py selects exactly one STT handler based on ModuleArguments.stt. You cannot run multiple speech-to-text engines in parallel within a single pipeline instance.

What happens if I pass the wrong argument class for my selected backend?

The pipeline ignores mismatched argument objects. If you set stt="faster-whisper" but only provide WhisperSTTHandlerArguments, the faster-whisper handler will use default values from FasterWhisperSTTHandlerArguments(). Always pair your stt value with the corresponding argument dataclass for predictable behavior.

Is Whisper MLX different from regular Whisper?

Yes. whisper-mlx uses LightningWhisperSTTHandler from src/speech_to_speech/STT/lightning_whisper_mlx_handler.py, which leverages Apple's MLX framework for optimized inference on Apple Silicon. It shares the same argument class as standard Whisper but routes to a different implementation optimized for Metal Performance Shaders.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}