# How to Use Faster Whisper as an STT Backend in Speech-to-Speech

> Integrate Faster Whisper as your STT backend for enhanced speech-to-speech performance. Learn how to set up this powerful tool easily for your projects. Improve your audio processing today.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-11

---

**TLDR:** Pass `--stt faster-whisper` to activate the backend, which instantiates `FasterWhisperSTTHandler` from [`src/speech_to_speech/STT/faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/faster_whisper_handler.py) and configures it via `FasterWhisperSTTHandlerArguments` defined in [`src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py).

The `huggingface/speech-to-speech` pipeline selects its speech-to-text component at runtime through a pluggable backend registry. Choosing **Faster Whisper** as your STT backend routes audio transcription through an optimized handler that supports quantized models and GPU acceleration. This guide explains the registration mechanism, configuration parameters, and practical usage patterns for deploying Faster Whisper in both server and local modes.

## How the Faster Whisper Backend Works

The integration relies on a factory pattern defined in the backend registry. When you specify `faster-whisper` as the STT provider, the system automatically imports and instantiates the correct handler class with your chosen model parameters.

### Backend Registration and Factory Pattern

In [`src/speech_to_speech/backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/backend_registry.py) (lines 31-42), the `faster-whisper` identifier is mapped to a factory function that imports `FasterWhisperSTTHandler` from `speech_to_speech.STT.faster_whisper_handler` (lines 33-38). The registry also binds the backend to its configuration dataclass, `FasterWhisperSTTHandlerArguments`, ensuring CLI flags are automatically validated and parsed.

### The FasterWhisperSTTHandler Lifecycle

The handler implements three core lifecycle methods in [`src/speech_to_speech/STT/faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/faster_whisper_handler.py):

- **`setup`** (lines 24-35): Initializes a `faster_whisper.WhisperModel` instance using the `model_name`, `device`, and `compute_type` supplied via arguments.
- **`process`** (lines 36-58): Receives VAD-extracted audio chunks, calls `self.model.transcribe`, concatenates resulting text segments, and yields `Transcription` messages for downstream LLM processing.
- **`cleanup`** (lines 61-63): Explicitly releases the Whisper model to free GPU memory.

## Configuration Options for Faster Whisper

The arguments class `FasterWhisperSTTHandlerArguments` in [`src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py) (lines 5-55) exposes the following CLI flags with their default values:

- `--faster_whisper_stt_model_name` – Model size (e.g., `tiny.en`, `small`, `large-v2`)
- `--faster_whisper_stt_device` – Compute device (`cpu`, `cuda`, or `auto`)
- `--faster_whisper_stt_compute_type` – Quantization type (`auto`, `int8`, `float16`)
- `--faster_whisper_stt_gen_max_new_tokens` – Token generation limit
- `--faster_whisper_stt_gen_beam_size` – Beam search width
- `--faster_whisper_stt_gen_return_timestamps` – Boolean for timestamp inclusion
- `--faster_whisper_stt_gen_task` – Task type (`transcribe` or `translate`)
- `--faster_whisper_stt_gen_language` – ISO language code (e.g., `en`, `fr`)

These flags are automatically added to the argument parser in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) using the `config_prefix` `faster_whisper_stt`.

## Usage Examples

You can configure Faster Whisper via command-line arguments, Python API, or JSON configuration files.

### CLI Server Mode

Deploy a realtime server with a specific Faster Whisper model on GPU:

```bash
speech-to-speech serve \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v2 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

```

### Local Loopback Mode

Run both the server and audio client on the same machine using automatic device detection:

```bash
speech-to-speech local \
    --stt faster-whisper \
    --faster_whisper_stt_model_name medium.en \
    --faster_whisper_stt_device auto

```

### Python API

Programmatically invoke the pipeline with backend-specific arguments:

```python
from speech_to_speech.s2s_pipeline import run_pipeline_command

run_pipeline_command(
    "serve",
    [
        "--stt", "faster-whisper",
        "--faster_whisper_stt_model_name", "large-v2",
        "--faster_whisper_stt_device", "cuda",
        "--faster_whisper_stt_compute_type", "float16",
    ],
)

```

### JSON Configuration File

Store parameters in a JSON file for reproducible deployments:

```json
{
    "stt": "faster-whisper",
    "faster_whisper_stt_model_name": "large-v2",
    "faster_whisper_stt_device": "cuda",
    "faster_whisper_stt_compute_type": "float16"
}

```

Execute with:

```bash
speech-to-speech serve config.json

```

## Summary

- The `faster-whisper` backend is registered in [`backend_registry.py`](https://github.com/huggingface/speech-to-speech/blob/main/backend_registry.py) with a dedicated factory function that imports `FasterWhisperSTTHandler`.
- Configuration is handled through `FasterWhisperSTTHandlerArguments`, exposing model name, device, compute type, and generation parameters.
- The handler implements `setup`, `process`, and `cleanup` methods to manage the Whisper model lifecycle and transcribe VAD-extracted audio chunks.
- Activation requires `--stt faster-whisper` followed by backend-specific flags prefixed with `faster_whisper_stt_`.
- Supported deployment modes include CLI server, local loopback, Python API, and JSON configuration.

## Frequently Asked Questions

### What Whisper models are supported when using Faster Whisper as an STT backend?

The backend accepts any model identifier supported by the underlying `faster-whisper` library, including `tiny.en`, `tiny`, `base.en`, `base`, `small.en`, `small`, `medium.en`, `medium`, `large-v1`, `large-v2`, and `large-v3`. Pass your chosen model via `--faster_whisper_stt_model_name`.

### How do I optimize Faster Whisper for GPU inference in the speech-to-speech pipeline?

Set `--faster_whisper_stt_device cuda` and `--faster_whisper_stt_compute_type float16` to enable half-precision GPU computation. For larger models that exceed VRAM, use `int8` quantization with `--faster_whisper_stt_compute_type int8` to reduce memory footprint while maintaining throughput.

### Can I use Faster Whisper with CPU-only setups?

Yes. Pass `--faster_whisper_stt_device cpu` and select `int8` quantization via `--faster_whisper_stt_compute_type int8` to optimize CPU inference speed. The `auto` setting will also automatically detect CPU availability if no GPU is present.

### Where is the actual transcription logic implemented in the source code?

The transcription logic resides in [`src/speech_to_speech/STT/faster_whisper_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/faster_whisper_handler.py) within the `process` method (lines 36-58). This method calls `self.model.transcribe` on incoming audio buffers, aggregates the resulting segments, and packages them into `Transcription` objects for the downstream language model handler.