# Supported STT Backends in speech-to-speech: Installation and Usage Guide

> Explore supported STT backends in huggingface speech-to-speech: Whisper, Faster-Whisper, Parakeet, and more. Learn how to easily install and use them for your projects.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**The huggingface/speech-to-speech library supports six distinct STT backends—`whisper`, `whisper-mlx`, `mlx-audio-whisper`, `faster-whisper`, `parakeet-tdt`, and `none`—each installable via optional pip extras defined in [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml).**

The **speech-to-speech** repository provides a modular architecture for real-time speech-to-text (STT) processing, with each backend designed for specific hardware and latency requirements. All backends are discovered at runtime through a centralized registry system that automatically wires CLI arguments and capability flags.

## How the STT Backend Registry Works

In [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py), the `STT_BACKENDS` dictionary maps backend names to `BackendInfo` objects containing three critical fields:

- **`kind`** — always `"stt"` for speech-to-text handlers
- **`create_handler`** — factory function returning the handler instance
- **capability flags** — such as `bypasses_transcription_notifier` (used by the `"none"` backend to skip transcription notification)

When you run `build_backend_registry()`, the system inspects this mapping and automatically generates CLI flags like `--stt <name>` and binds the corresponding argument class (e.g., `WhisperSTTHandlerArguments`).

## Complete List of Supported STT Backends

| Backend (CLI flag) | Handler Class | Best For | Platform |
|-------------------|---------------|----------|----------|
| `none` | — | Audio-only pipelines, external transcription | All |
| `whisper` | `WhisperSTTHandler` in [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py) | General-purpose CPU transcription | All |
| `whisper-mlx` | `LightningWhisperSTTHandler` in [`lightning_whisper_mlx_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/lightning_whisper_mlx_handler.py) | Fast macOS inference | macOS (Apple Silicon) |
| `mlx-audio-whisper` | `MLXAudioWhisperSTTHandler` in [`mlx_audio_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_audio_whisper_handler.py) | High-performance GPU/CPU via MLX | AMD/Apple GPUs |
| `faster-whisper` | `FasterWhisperSTTHandler` in [`faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/faster_whisper_handler.py) | Low-latency CUDA inference | NVIDIA GPUs |
| `parakeet-tdt` | `ParakeetTDTSTTHandler` in [`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py) | Real-time streaming, low memory | Default on all platforms |

The **default STT backend** is `parakeet-tdt`, which automatically selects `nano-parakeet` on CPU/CUDA or `mlx-community/parakeet-tdt-0.6b-v3` on macOS MPS. For ultra-low latency, add `--progressive_streaming` to enable chunked output via `SmartProgressiveStreamingHandler` in [`smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_progressive_streaming.py).

> **Note:** An archived `moonshine` handler exists in [`archive/STT/moonshine_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/archive/STT/moonshine_handler.py) but is **not registered** in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) and therefore unsupported.

## Installing STT Backends

The base package includes `torch`, `transformers`, and `accelerate`. Backend-specific dependencies require **optional extras**:

```bash

# Base installation (no STT backend)

pip install speech-to-speech

# Lightning-Whisper-MLX for Apple Silicon

pip install "speech-to-speech[whisper-mlx]"

# Faster-Whisper for NVIDIA CUDA

pip install "speech-to-speech[faster-whisper]"

# Parakeet-TDT (nano-parakeet on Linux/Windows, MLX variant on macOS)

pip install "speech-to-speech[parakeet]"

# MLX-Audio Whisper for cross-platform GPU acceleration

pip install "speech-to-speech[mlx-audio-whisper]"

```

To install **all supported STT backends** at once:

```bash
pip install "speech-to-speech[whisper-mlx,faster-whisper,parakeet,mlx-audio-whisper]"

```

The [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml) conditional logic ensures `whisper-mlx` only installs on Darwin (macOS), while `parakeet` pulls `nano-parakeet>=0.2.0` on non-Darwin systems.

## Using STT Backends: CLI Examples

All backend-specific arguments are auto-generated from their handler argument classes:

```bash

# Faster-Whisper on CUDA with custom model

speech-to-speech serve --stt faster-whisper \
    --faster_whisper_model_name openai/whisper-large-v3 \
    --faster_whisper_device cuda

# Parakeet-TDT on Apple Silicon with MLX-optimized model

speech-to-speech serve --stt parakeet-tdt \
    --parakeet_tdt_model_name mlx-community/parakeet-tdt-0.6b-v3

# Parakeet-TDT with progressive streaming for live captioning

speech-to-speech serve --stt parakeet-tdt --progressive_streaming

# CPU-only Whisper (no extras required)

speech-to-speech serve --stt whisper \
    --whisper_model_name distil-whisper/distil-large-v3

```

## Programmatic STT Backend Usage

Access the registry directly for custom pipelines:

```python
from speech_to_speech.backend_registry import STT_BACKENDS
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

# Select and configure MLX-Audio Whisper

stt_info = STT_BACKENDS["mlx-audio-whisper"]
stt_handler = stt_info.create_handler(
    model_name="mlx-community/whisper-large-v3-turbo",
    language="en"
)

# Build pipeline with STT-only (no LLM/TTS)

pipeline = SpeechToSpeechPipeline(
    stt_backend=stt_info,
    llm_backend=STT_BACKENDS["none"],
)

# Transcribe

text = pipeline.transcribe("recording.wav")
print(f"Transcription: {text}")

```

## Adding Custom STT Backends

To extend supported STT backends:

1. Create a subclass of `BaseSTTHandler` in `src/speech_to_speech/STT/`
2. Define CLI arguments in [`src/speech_to_speech/arguments_classes/your_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/your_stt_arguments.py)
3. Register in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py): add entry to `STT_BACKENDS` with `kind="stt"`
4. Add optional extra to [`pyproject.toml`](https://github.com/huggingface/speech-to-speech/blob/main/pyproject.toml) if external packages are required

## Summary

- **Six official STT backends** are registered in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py): `whisper`, `whisper-mlx`, `mlx-audio-whisper`, `faster-whisper`, `parakeet-tdt` (default), and `none`
- **Installation** uses pip extras: `[whisper-mlx]`, `[faster-whisper]`, `[parakeet]`, `[mlx-audio-whisper]`
- **Platform restrictions**: `whisper-mlx` is macOS-only; `parakeet` auto-selects MLX on Darwin
- **Progressive streaming** via `--progressive_streaming` flag on `parakeet-tdt` enables chunked real-time output
- **Handler classes** are located in `src/speech_to_speech/STT/` with corresponding argument classes in `arguments_classes/`

## Frequently Asked Questions

### What is the default STT backend in speech-to-speech?

The default is `parakeet-tdt`. According to the [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) source code, this backend automatically selects `nano-parakeet` on CPU/CUDA systems or `mlx-community/parakeet-tdt-0.6b-v3` when running on macOS with MPS. It is optimized for low memory footprint and real-time streaming.

### Can I use speech-to-speech on Mac without NVIDIA GPU?

Yes. Install the `whisper-mlx` or `mlx-audio-whisper` extras for Apple Silicon acceleration, or use the default `parakeet-tdt` which automatically uses MLX-optimized models on Darwin. The `faster-whisper` backend requires CUDA and will not benefit macOS users.

### Why would I use the `none` STT backend?

The `none` backend sets `bypasses_transcription_notifier=True` and performs no transcription. Use it when audio passes through without text conversion, or when transcription is handled externally. It requires zero additional dependencies.

### How do I enable the fastest possible transcription latency?

Use `parakeet-tdt` with `--progressive_streaming` flag. This activates `SmartProgressiveStreamingHandler` in [`smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_progressive_streaming.py), which yields partial transcriptions chunk-by-chunk rather than waiting for complete utterances. For NVIDIA GPUs, `faster-whisper` with CUDA provides the lowest single-pass latency.