# Swapping STT Backends: Parakeet TDT vs Whisper vs Faster Whisper in Hugging Face Speech-to-Speech

> Easily swap STT backends like Parakeet TDT, Whisper, and Faster Whisper in Hugging Face Speech-to-Speech. Discover flexible speech processing with a single flag.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: comparison
- Published: 2026-08-03

---

**The Hugging Face Speech-to-Speech library treats every Speech-to-Text engine as an interchangeable handler, allowing you to swap between Parakeet TDT, Whisper, and Faster Whisper by changing a single `--stt` flag without modifying pipeline code.** Swapping STT backends in this repository leverages a unified `BaseSTTHandler` interface where the dispatcher in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) instantiates the correct implementation based on the `module_kwargs.stt` field. Each backend offers distinct trade-offs in streaming capability, device optimization, and language support while maintaining full compatibility with the real-time speech pipeline.

## Unified Handler Architecture

Every STT backend inherits from `BaseSTTHandler` and implements standardized `setup`, `process`, and optional `cleanup` methods. This abstraction enables the pipeline construction logic in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 868-890) to treat all engines identically.

The dispatcher selects the concrete class using the `module_kwargs.stt` value:

- **`parakeet-tdt`** → `ParakeetTDTSTTHandler`
- **`whisper`** → `WhisperSTTHandler`  
- **`faster-whisper`** → `FasterWhisperSTTHandler`

The system unpacks the corresponding argument dataclass—`ParakeetTDTSTTHandlerArguments`, `WhisperSTTHandlerArguments`, or `FasterWhisperSTTHandlerArguments`—and forwards these parameters as `setup_kwargs` to the handler constructor.

## Backend Comparison

### Parakeet TDT

**`ParakeetTDTSTTHandler`** provides the only implementation with **live progressive transcription** in the library. It integrates `SmartProgressiveStreamingHandler` to emit `PartialTranscription` objects while audio is still arriving, enabling real-time UI updates when `module_kwargs.enable_live_transcription` is set to `true` (lines 60-73 and 146-151 of [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py)).

Device selection is automatic: the handler detects macOS (`platform == "darwin"`) to use MPS via `mlx-audio`, otherwise CUDA via `nano-parakeet`, with CPU fallback (lines 31-48). Language support covers 25 European languages via auto-detection using `lingua-py` when the language parameter is `None` (lines 79-104).

### Whisper

**`WhisperSTTHandler`** wraps 🤗 Transformers models and requires explicit device management through the `stt_device` argument (defaulting to `cuda`). Unlike Parakeet TDT, it only returns final `Transcription` objects after the full utterance decodes, making it unsuitable for streaming use cases requiring partial results.

Language handling extracts the language token from the first generated token, falling back to the last known language if the detected code is unsupported (defined in the `SUPPORTED_LANGUAGES` constant in [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py)).

### Faster Whisper

**`FasterWhisperSTTHandler`** leverages the `faster-whisper` library (CTranslate2 backend) for optimized inference. It delegates device detection to the `WhisperModel` constructor and does not perform explicit language detection; instead, language is inferred from the model name (e.g., `tiny.en` implies English).

This handler returns a single `Transcription` after full-audio decode, similar to standard Whisper, but typically achieves lower latency through quantization and batching optimizations.

## How to Swap STT Backends

### Command-Line Interface

Change the `--stt` flag to switch implementations:

```bash

# Parakeet TDT with live transcription

python -m speech_to_speech.main \
    --stt parakeet-tdt \
    --enable_live_transcription true \
    --live_transcription_update_interval 0.3

# Standard Whisper

python -m speech_to_speech.main \
    --stt whisper \
    --stt_model_name distil-whisper/distil-large-v3 \
    --stt_device cuda

# Faster Whisper optimized

python -m speech_to_speech.main \
    --stt faster-whisper \
    --faster_whisper_stt_model_name tiny.en \
    --faster_whisper_stt_device auto

```

### Programmatic Configuration

Instantiate the pipeline with specific argument dataclasses:

```python
from speech_to_speech.pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes import (
    ModuleArguments,
    ParakeetTDTSTTHandlerArguments,
    WhisperSTTHandlerArguments,
    FasterWhisperSTTHandlerArguments,
)

def make_pipeline(backend: str):
    mod_args = ModuleArguments(stt=backend, enable_live_transcription=True)
    
    if backend == "parakeet-tdt":
        stt_args = ParakeetTDTSTTHandlerArguments(
            parakeet_tdt_device="auto",
            parakeet_tdt_compute_type="float16"
        )
    elif backend == "whisper":
        stt_args = WhisperSTTHandlerArguments(
            stt_model_name="distil-whisper/distil-large-v3",
            stt_device="cuda",
            stt_torch_dtype="float16"
        )
    else:  # faster-whisper

        stt_args = FasterWhisperSTTHandlerArguments(
            faster_whisper_stt_model_name="small.en",
            faster_whisper_stt_device="auto"
        )
    
    return SpeechToSpeechPipeline(module_kwargs=mod_args, stt_kwargs=stt_args)

pipeline = make_pipeline("parakeet-tdt")

```

## Implementation Deep Dive

### Device Strategy Differences

Each backend handles hardware acceleration differently according to [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) (lines 32-40):

- **Parakeet TDT**: Automatic detection prefers MPS on macOS, then CUDA, then CPU
- **Whisper**: Respects explicit `stt_device` parameter (default `cuda`)
- **Faster Whisper**: Uses `faster-whisper`'s internal auto-detection via the `device` parameter

### Live Transcription Capability

Only `ParakeetTDTSTTHandler` supports progressive streaming. When `enable_live_transcription` is true, the handler initializes `SmartProgressiveStreamingHandler` to chunk and emit partial results. Whisper-based handlers wait for silence detection before returning the complete transcription.

### Language Detection Mechanisms

- **Parakeet TDT**: Uses `lingua-py` library on the transcript text when `language` is `None`
- **Whisper**: Extracts language from the first generated token; falls back to previous language if unsupported
- **Faster Whisper**: No runtime detection; language is determined by the model file suffix (e.g., `.en` for English)

## Summary

- The Speech-to-Speech library abstracts STT engines through the `BaseSTTHandler` interface, enabling backend swaps via a single configuration change.
- **Parakeet TDT** offers the only live progressive transcription with automatic device selection and 25-language auto-detection via `lingua-py`.
- **Whisper** provides broad model compatibility through 🤗 Transformers but only returns final transcriptions after full utterance processing.
- **Faster Whisper** delivers optimized CTranslate2 inference but requires language-specific model selection and lacks partial result streaming.
- The dispatcher in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 868-890) routes to the correct handler based on `module_kwargs.stt` values: `parakeet-tdt`, `whisper`, or `faster-whisper`.

## Frequently Asked Questions

### Can I switch between STT backends without changing my pipeline code?

Yes. The library implements a handler factory pattern where `get_stt_handler` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) instantiates the correct class based on the `module_kwargs.stt` field. You only need to change the `--stt` CLI flag or the corresponding configuration field to swap between Parakeet TDT, Whisper, and Faster Whisper implementations.

### Why does only Parakeet TDT support live transcription?

Parakeet TDT integrates `SmartProgressiveStreamingHandler` to emit `PartialTranscription` objects while audio is still arriving, as implemented in [`parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/parakeet_tdt_handler.py) lines 60-73. The Whisper and Faster Whisper handlers buffer the entire utterance until silence is detected, then return a single complete `Transcription` object, making them unsuitable for real-time streaming updates.

### How does automatic language detection work across backends?

Parakeet TDT uses the `lingua-py` library to analyze transcript text when no language is specified. Whisper extracts the language token from the model's first generated output and validates it against the `SUPPORTED_LANGUAGES` list. Faster Whisper does not perform runtime detection; you must load a language-specific model (e.g., `tiny.en` for English) to ensure correct transcription.

### Which backend should I choose for Apple Silicon (MPS) acceleration?

Use **Parakeet TDT**, which automatically detects macOS and routes to `mlx-audio` for MPS acceleration. The Whisper handler requires explicit device specification and uses PyTorch, while Faster Whisper delegates to CTranslate2 which may have limited MPS optimization compared to the MLX implementation in Parakeet TDT.