How to Select Different STT Backends for Speech-to-Speech: A Complete Guide

Use the --stt flag with any registered backend name (e.g., whisper, faster-whisper, mlx-audio-whisper) or construct a BackendSelection object programmatically to switch speech-to-text engines in the Hugging Face Speech-to-Speech pipeline.

The Hugging Face Speech-to-Speech (S2S) framework treats every pipeline component—including speech-to-text—as a pluggable backend. Whether you need faster inference, GPU acceleration on Apple Silicon, or specialized models for non-English languages, switching STT backends requires no code changes beyond a command-line flag or constructor argument.

Where STT Backends Are Registered

All supported STT backends are centralized in src/speech_to_speech/backend_registry.py. The STT_BACKENDS dictionary maps short names to BackendInfo objects containing the module path and configuration prefix:

STT_BACKENDS = {
    "whisper": BackendInfo(
        module="speech_to_speech.STT.whisper_stt_handler",
        config_prefix="stt",
        kind="stt",
    ),
    "whisper-mlx": BackendInfo(
        module="speech_to_speech.STT.lightning_whisper_mlx_handler",
        config_prefix="stt",
        kind="stt",
    ),
    "mlx-audio-whisper": BackendInfo(
        module="speech_to_speech.STT.mlx_audio_whisper_handler",
        config_prefix="mlx_audio_whisper_stt",
        kind="stt",
    ),
    "faster-whisper": BackendInfo(
        module="speech_to_speech.STT.faster_whisper_handler",
        config_prefix="faster_whisper_stt",
        kind="stt",
    ),
    "parakeet-tdt": BackendInfo(
        module="speech_to_speech.STT.parakeet_tdt_handler",
        config_prefix="parakeet_tdt",
        kind="stt",
    ),
    "paraformer": BackendInfo(
        module="speech_to_speech.STT.paraformer_handler",
        config_prefix="paraformer_stt",
        kind="stt",
    ),
}

Each entry specifies:

• module: Python import path to the handler class
• config_prefix: Namespace for CLI arguments specific to that backend
• kind: Backend category ("stt" for speech-to-text)

How Runtime Selection Works

The selection mechanism spans two core files: backend_registry.py for definitions and src/speech_to_speech/s2s_pipeline.py for orchestration.

Command-Line Parsing

In s2s_pipeline.py, the argument parser constrains --stt to valid backend keys:

_pre.add_argument(
    "--stt",
    choices=tuple(STT_BACKENDS),
    help="Select the STT backend (default is the module default)",
)

Pipeline Instantiation

After parsing, the selected name becomes a BackendSelection object:

stt_backend = BackendSelection(kind="stt", name=_stt_name)

This object feeds into create_backend_handler(stt_backend, stt_context), which imports and instantiates the concrete handler (e.g., WhisperSTTHandler from STT/whisper_stt_handler.py). The handler produces STTOutItem objects placed on stt_output_queue. Depending on spec.capabilities.bypasses_transcription_notifier, the pipeline either routes through a TranscriptionNotifier or sends raw text directly to the LLM queue.

Selecting STT Backends from the Command Line

Pass the backend name to --stt, followed by backend-specific arguments prefixed according to the registry entry:

# Whisper (default, CPU-friendly)

speech-to-speech serve --stt whisper --language en

# MLX-Audio Whisper: GPU-accelerated on Apple Silicon

speech-to-speech serve --stt mlx-audio-whisper --device mps

# Faster-Whisper with German language model

speech-to-speech serve \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v3 \
    --faster_whisper_stt_gen_language de

# Parakeet TDT with automatic device selection

speech-to-speech serve \
    --stt parakeet-tdt \
    --parakeet_tdt_device auto

# Paraformer for streaming Mandarin or other CJK languages

speech-to-speech serve \
    --stt paraformer \
    --paraformer_stt_model_name damo/speech_paraformer-large_asr_nat-zh-cn-16k-common-vocab8404-pytorch

Backend-specific flags live in src/speech_to_speech/arguments_classes/ (e.g., faster_whisper_stt_arguments.py, paraformer_stt_arguments.py).

Selecting STT Backends Programmatically

For embedded applications or custom orchestration, construct BackendSelection directly:

from speech_to_speech.backend_registry import BackendSelection, STT_BACKENDS
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

# Select the "paraformer" backend

stt_backend = BackendSelection(kind="stt", name="paraformer")

pipeline = SpeechToSpeechPipeline(
    stt_backend=stt_backend,
    # ... additional required backends: llm_backend, tts_backend, etc.

)

pipeline.run()

Inspect the selection for debugging or dynamic configuration:

print(stt_backend.spec.module)          # 'speech_to_speech.STT.paraformer_handler'

print(stt_backend.spec.config_prefix)   # 'paraformer_stt'

Adding a Custom STT Backend

Extend the framework with proprietary or experimental models in three steps:

Step 1: Implement the Handler

Subclass BaseSTTHandler in a new file under src/speech_to_speech/STT/:

# src/speech_to_speech/STT/my_custom_handler.py

from speech_to_speech.STT.base_stt_handler import BaseSTTHandler

class MyCustomSTTHandler(BaseSTTHandler):
    def __init__(self, args):
        super().__init__(args)
        # Initialize your ASR model here

        self.model = load_model(args.my_custom_stt_model_path)

    async def process(self, audio_chunk: bytes) -> str:
        """Transcribe audio chunk to text."""
        text = self.model.transcribe(audio_chunk)
        return text

Step 2: Register the Backend

Add an entry to STT_BACKENDS in backend_registry.py:

STT_BACKENDS["my-custom"] = BackendInfo(
    module="speech_to_speech.STT.my_custom_handler",
    config_prefix="my_custom_stt",
    kind="stt",
)

Step 3: (Optional) Define CLI Arguments

Create src/speech_to_speech/arguments_classes/my_custom_stt_arguments.py:

from dataclasses import dataclass

@dataclass
class MyCustomSTTArguments:
    my_custom_stt_model_path: str = "models/my-model"
    my_custom_stt_language: str = "auto"

The new backend is immediately available via --stt my-custom.

Available STT Backends Reference

Backend	Best For	Key File
whisper	General-purpose, CPU inference	STT/whisper_stt_handler.py
whisper-mlx	Apple Silicon, Lightning-fast inference	STT/lightning_whisper_mlx_handler.py
mlx-audio-whisper	Apple Silicon, unified audio toolkit	STT/mlx_audio_whisper_handler.py
faster-whisper	Production throughput, CTranslate2	STT/faster_whisper_handler.py
parakeet-tdt	NVIDIA GPU-optimized streaming	STT/parakeet_tdt_handler.py
paraformer	Streaming Mandarin/CJK recognition	STT/paraformer_handler.py

Summary

• Registry location: All STT backends are defined in src/speech_to_speech/backend_registry.py as STT_BACKENDS entries
• CLI selection: Use --stt <backend_name> with backend-specific prefixed arguments
• Programmatic selection: Pass BackendSelection(kind="stt", name="...") to SpeechToSpeechPipeline
• Extensibility: Implement BaseSTTHandler, register in STT_BACKENDS, optionally add argument classes

Frequently Asked Questions

How do I find which STT backends are currently installed?

Check the STT_BACKENDS dictionary keys in src/speech_to_speech/backend_registry.py or run speech-to-speech serve --help to see the --stt choices. The repository does not support runtime enumeration of handler modules; the registry hardcodes available options.

Can I use multiple STT backends simultaneously in one pipeline?

No. The SpeechToSpeechPipeline accepts a single stt_backend argument. To A/B test backends, instantiate separate pipeline objects or implement a custom handler that internally multiplexes between multiple ASR models based on audio characteristics.

What happens if I specify an STT backend without its required dependencies?

The handler import fails during create_backend_handler() with a ModuleNotFoundError or ImportError. Install backend-specific requirements (e.g., pip install faster-whisper for the faster-whisper backend) before selection. Dependencies are not enforced at the registry level.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}