# How to Use Whisper as an STT Backend in the Speech-to-Speech Pipeline

> Learn how to use Whisper as an STT backend for your speech-to-speech pipeline. Configure model parameters and generation settings easily via the CLI for efficient transcription.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**To use Whisper as an STT backend, pass `--stt whisper` to the CLI and configure model parameters via `--stt_model_name`, `--stt_device`, and generation settings prefixed with `--stt_gen_`.**

The `huggingface/speech-to-speech` repository implements a modular voice-agent pipeline (**VAD → STT → LLM → TTS**) where each stage is a pluggable backend. When you use Whisper as an STT backend, the system dynamically loads `WhisperSTTHandler` from [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py) to process audio chunks into text transcriptions.

## Architecture Overview

The Speech-to-Speech pipeline uses a registry pattern to map backend names to handler implementations. When you configure Whisper as your STT provider, the system resolves the `whisper` key in the `STT_BACKENDS` registry defined in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 97-108).

The data flow follows this sequence:

1. **Argument Parsing** – The `parse_arguments` function in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) reads the `--stt` flag and instantiates `WhisperSTTHandlerArguments` from [`src/speech_to_speech/arguments_classes/whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/whisper_stt_arguments.py).

2. **Configuration Normalization** – The `BackendSelection.normalize` method converts the dataclass into a flat dictionary of kwargs, merging any generation parameters prefixed with `gen_`.

3. **Handler Instantiation** – The `create_backend_handler` factory dynamically imports `WhisperSTTHandler` and passes the normalized configuration.

4. **Runtime Execution** – The handler receives audio chunks from the VAD stage, preprocesses them using `AutoProcessor`, runs `model.generate`, and pushes a `Transcription` object downstream to the LLM.

The handler supports auto-detection of spoken languages when you specify `--language auto`, or you can constrain recognition to a specific language code (defined in the handler's `SUPPORTED_LANGUAGES` constant).

## Configuration Options

Whisper-specific settings are exposed through CLI flags defined in `WhisperSTTHandlerArguments`. All generation parameters for the underlying `transformers` model use the `--stt_gen_*` prefix.

**Key parameters:**

- `--stt_model_name` – Hugging Face Hub model ID (e.g., `openai/whisper-large-v3` or `distil-whisper/distil-large-v3`)
- `--stt_device` – Compute device (`cuda`, `cpu`, or `mps`)
- `--stt_torch_dtype` – Model precision (`float16`, `bfloat16`, or `float32`)
- `--stt_compile_mode` – Torch compile optimization mode
- `--language` – Language code (e.g., `en`, `fr`) or `auto` for auto-detection
- `--stt_gen_max_new_tokens` – Maximum tokens per generation
- `--stt_gen_num_beams` – Beam search width
- `--stt_gen_temperature` – Sampling temperature

## Running Whisper STT

### Server Mode with Whisper

Deploy the Realtime server using Whisper for speech-to-text transcription:

```bash

# Install optional Whisper dependencies

pip install "speech-to-speech[whisper]"

# Start server with Whisper large-v3 on GPU

speech-to-speech serve \
    --stt whisper \
    --stt_model_name openai/whisper-large-v3 \
    --stt_device cuda \
    --stt_torch_dtype float16 \
    --stt_gen_max_new_tokens 128 \
    --language auto

```

This command initializes the `WhisperSTTHandler` with `AutoModelForSpeechSeq2Seq.from_pretrained`, executes a warmup forward pass to prime CUDA graphs, and listens for WebSocket connections.

Connect a client to stream microphone audio:

```bash
speech-to-speech talk \
    --url ws://127.0.0.1:8765/v1/realtime \
    --model local

```

### Local Mode

Run both the server and client in a single process for local testing:

```bash
speech-to-speech local \
    --stt whisper \
    --stt_model_name distil-whisper/distil-large-v3 \
    --stt_device cpu \
    --language en

```

Local mode instantiates the handler in the main process, bypassing the network layer while maintaining the same `STT_BACKENDS` registry resolution logic found in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) (lines 70-77).

### Custom Generation Settings

Tune the transcription behavior by passing generation kwargs directly to the underlying model:

```bash
speech-to-speech serve \
    --stt whisper \
    --stt_gen_max_new_tokens 256 \
    --stt_gen_temperature 0.0 \
    --stt_gen_num_beams 4

```

The `prepare_model_inputs` method in [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py) builds the `gen_kwargs` dictionary from these flags and passes it to `model.generate()`.

## Implementation Details

The `WhisperSTTHandler` class in [`src/speech_to_speech/STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/whisper_stt_handler.py) handles model lifecycle and inference:

- **Model Loading** – Uses `AutoProcessor.from_pretrained` and `AutoModelForSpeechSeq2Seq.from_pretrained` with device mapping and dtype conversion
- **Warmup** – The `warmup` method runs a dummy forward pass to initialize Torch-Compile caches or CUDA graphs before processing real-time audio
- **Processing** – The `process` method implements the handler interface, receiving audio chunks and returning `Transcription` dataclasses

The backend registration in [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) wires the `whisper` string key to this handler class, enabling runtime selection via the `--stt` CLI argument.

## Summary

- Use `--stt whisper` to select the Whisper backend when running `speech-to-speech serve` or `speech-to-speech local`
- Configure the model via `--stt_model_name`, `--stt_device`, and `--stt_torch_dtype` flags
- Pass generation parameters using the `--stt_gen_*` prefix to control beam search, temperature, and token limits
- The handler auto-detects language with `--language auto` or accepts specific language codes
- Source implementations reside in [`whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/whisper_stt_handler.py) and register via [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py)

## Frequently Asked Questions

### What Whisper models are supported?

You can use any Whisper checkpoint available on the Hugging Face Hub, including `openai/whisper-large-v3`, `openai/whisper-base`, and distilled variants like `distil-whisper/distil-large-v3`. Pass the model ID to `--stt_model_name`.

### How do I enable language auto-detection?

Add `--language auto` to your CLI command. The handler uses the model's built-in language classification capabilities to identify the spoken language from the audio content. Alternatively, specify a language code (e.g., `--language en`) to constrain recognition.

### Can I use Torch compile with Whisper STT?

Yes. Pass `--stt_compile_mode default` or `--stt_compile_mode max-autotune` to enable Torch compilation. The `warmup` method in `WhisperSTTHandler` runs a dummy inference pass to compile the graph before processing real audio chunks, preventing compilation latency during live transcription.

### Why is there a warmup phase when starting the server?

The `warmup` method executes a forward pass with dummy inputs to initialize CUDA memory pools and Torch-Compile caches. This prevents cold-start latency when the first real audio arrives from the VAD stage, ensuring consistent real-time performance for the voice pipeline.