Supported STT Backends in speech-to-speech: Installation and Usage Guide

The huggingface/speech-to-speech library supports six distinct STT backends—whisper, whisper-mlx, mlx-audio-whisper, faster-whisper, parakeet-tdt, and none—each installable via optional pip extras defined in pyproject.toml.

The speech-to-speech repository provides a modular architecture for real-time speech-to-text (STT) processing, with each backend designed for specific hardware and latency requirements. All backends are discovered at runtime through a centralized registry system that automatically wires CLI arguments and capability flags.

How the STT Backend Registry Works

In src/speech_to_speech/backend_registry.py, the STT_BACKENDS dictionary maps backend names to BackendInfo objects containing three critical fields:

• kind — always "stt" for speech-to-text handlers
• create_handler — factory function returning the handler instance
• capability flags — such as bypasses_transcription_notifier (used by the "none" backend to skip transcription notification)

When you run build_backend_registry(), the system inspects this mapping and automatically generates CLI flags like --stt <name> and binds the corresponding argument class (e.g., WhisperSTTHandlerArguments).

Complete List of Supported STT Backends

Backend (CLI flag)	Handler Class	Best For	Platform
none	—	Audio-only pipelines, external transcription	All
whisper	WhisperSTTHandler in whisper_stt_handler.py	General-purpose CPU transcription	All
whisper-mlx	LightningWhisperSTTHandler in lightning_whisper_mlx_handler.py	Fast macOS inference	macOS (Apple Silicon)
mlx-audio-whisper	MLXAudioWhisperSTTHandler in mlx_audio_whisper_handler.py	High-performance GPU/CPU via MLX	AMD/Apple GPUs
faster-whisper	FasterWhisperSTTHandler in faster_whisper_handler.py	Low-latency CUDA inference	NVIDIA GPUs
parakeet-tdt	ParakeetTDTSTTHandler in parakeet_tdt_handler.py	Real-time streaming, low memory	Default on all platforms

The default STT backend is parakeet-tdt, which automatically selects nano-parakeet on CPU/CUDA or mlx-community/parakeet-tdt-0.6b-v3 on macOS MPS. For ultra-low latency, add --progressive_streaming to enable chunked output via SmartProgressiveStreamingHandler in smart_progressive_streaming.py.

Note: An archived moonshine handler exists in archive/STT/moonshine_handler.py but is not registered in backend_registry.py and therefore unsupported.

Installing STT Backends

The base package includes torch, transformers, and accelerate. Backend-specific dependencies require optional extras:

# Base installation (no STT backend)

pip install speech-to-speech

# Lightning-Whisper-MLX for Apple Silicon

pip install "speech-to-speech[whisper-mlx]"

# Faster-Whisper for NVIDIA CUDA

pip install "speech-to-speech[faster-whisper]"

# Parakeet-TDT (nano-parakeet on Linux/Windows, MLX variant on macOS)

pip install "speech-to-speech[parakeet]"

# MLX-Audio Whisper for cross-platform GPU acceleration

pip install "speech-to-speech[mlx-audio-whisper]"

To install all supported STT backends at once:

pip install "speech-to-speech[whisper-mlx,faster-whisper,parakeet,mlx-audio-whisper]"

The pyproject.toml conditional logic ensures whisper-mlx only installs on Darwin (macOS), while parakeet pulls nano-parakeet>=0.2.0 on non-Darwin systems.

Using STT Backends: CLI Examples

All backend-specific arguments are auto-generated from their handler argument classes:

# Faster-Whisper on CUDA with custom model

speech-to-speech serve --stt faster-whisper \
    --faster_whisper_model_name openai/whisper-large-v3 \
    --faster_whisper_device cuda

# Parakeet-TDT on Apple Silicon with MLX-optimized model

speech-to-speech serve --stt parakeet-tdt \
    --parakeet_tdt_model_name mlx-community/parakeet-tdt-0.6b-v3

# Parakeet-TDT with progressive streaming for live captioning

speech-to-speech serve --stt parakeet-tdt --progressive_streaming

# CPU-only Whisper (no extras required)

speech-to-speech serve --stt whisper \
    --whisper_model_name distil-whisper/distil-large-v3

Programmatic STT Backend Usage

Access the registry directly for custom pipelines:

from speech_to_speech.backend_registry import STT_BACKENDS
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

# Select and configure MLX-Audio Whisper

stt_info = STT_BACKENDS["mlx-audio-whisper"]
stt_handler = stt_info.create_handler(
    model_name="mlx-community/whisper-large-v3-turbo",
    language="en"
)

# Build pipeline with STT-only (no LLM/TTS)

pipeline = SpeechToSpeechPipeline(
    stt_backend=stt_info,
    llm_backend=STT_BACKENDS["none"],
)

# Transcribe

text = pipeline.transcribe("recording.wav")
print(f"Transcription: {text}")

Adding Custom STT Backends

To extend supported STT backends:

1. Create a subclass of BaseSTTHandler in src/speech_to_speech/STT/
2. Define CLI arguments in src/speech_to_speech/arguments_classes/your_stt_arguments.py
3. Register in backend_registry.py: add entry to STT_BACKENDS with kind="stt"
4. Add optional extra to pyproject.toml if external packages are required

Summary

• Six official STT backends are registered in backend_registry.py: whisper, whisper-mlx, mlx-audio-whisper, faster-whisper, parakeet-tdt (default), and none
• Installation uses pip extras: [whisper-mlx], [faster-whisper], [parakeet], [mlx-audio-whisper]
• Platform restrictions: whisper-mlx is macOS-only; parakeet auto-selects MLX on Darwin
• Progressive streaming via --progressive_streaming flag on parakeet-tdt enables chunked real-time output
• Handler classes are located in src/speech_to_speech/STT/ with corresponding argument classes in arguments_classes/

Frequently Asked Questions

What is the default STT backend in speech-to-speech?

The default is parakeet-tdt. According to the backend_registry.py source code, this backend automatically selects nano-parakeet on CPU/CUDA systems or mlx-community/parakeet-tdt-0.6b-v3 when running on macOS with MPS. It is optimized for low memory footprint and real-time streaming.

Can I use speech-to-speech on Mac without NVIDIA GPU?

Yes. Install the whisper-mlx or mlx-audio-whisper extras for Apple Silicon acceleration, or use the default parakeet-tdt which automatically uses MLX-optimized models on Darwin. The faster-whisper backend requires CUDA and will not benefit macOS users.

Why would I use the none STT backend?

The none backend sets bypasses_transcription_notifier=True and performs no transcription. Use it when audio passes through without text conversion, or when transcription is handled externally. It requires zero additional dependencies.

How do I enable the fastest possible transcription latency?

Use parakeet-tdt with --progressive_streaming flag. This activates SmartProgressiveStreamingHandler in smart_progressive_streaming.py, which yields partial transcriptions chunk-by-chunk rather than waiting for complete utterances. For NVIDIA GPUs, faster-whisper with CUDA provides the lowest single-pass latency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}