How to Use Faster Whisper as an STT Backend in Speech-to-Speech

TLDR: Pass --stt faster-whisper to activate the backend, which instantiates FasterWhisperSTTHandler from src/speech_to_speech/STT/faster_whisper_handler.py and configures it via FasterWhisperSTTHandlerArguments defined in src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py.

The huggingface/speech-to-speech pipeline selects its speech-to-text component at runtime through a pluggable backend registry. Choosing Faster Whisper as your STT backend routes audio transcription through an optimized handler that supports quantized models and GPU acceleration. This guide explains the registration mechanism, configuration parameters, and practical usage patterns for deploying Faster Whisper in both server and local modes.

How the Faster Whisper Backend Works

The integration relies on a factory pattern defined in the backend registry. When you specify faster-whisper as the STT provider, the system automatically imports and instantiates the correct handler class with your chosen model parameters.

Backend Registration and Factory Pattern

In src/speech_to_speech/backend_registry.py (lines 31-42), the faster-whisper identifier is mapped to a factory function that imports FasterWhisperSTTHandler from speech_to_speech.STT.faster_whisper_handler (lines 33-38). The registry also binds the backend to its configuration dataclass, FasterWhisperSTTHandlerArguments, ensuring CLI flags are automatically validated and parsed.

The FasterWhisperSTTHandler Lifecycle

The handler implements three core lifecycle methods in src/speech_to_speech/STT/faster_whisper_handler.py:

• setup (lines 24-35): Initializes a faster_whisper.WhisperModel instance using the model_name, device, and compute_type supplied via arguments.
• process (lines 36-58): Receives VAD-extracted audio chunks, calls self.model.transcribe, concatenates resulting text segments, and yields Transcription messages for downstream LLM processing.
• cleanup (lines 61-63): Explicitly releases the Whisper model to free GPU memory.

Configuration Options for Faster Whisper

The arguments class FasterWhisperSTTHandlerArguments in src/speech_to_speech/arguments_classes/faster_whisper_stt_arguments.py (lines 5-55) exposes the following CLI flags with their default values:

• --faster_whisper_stt_model_name – Model size (e.g., tiny.en, small, large-v2)
• --faster_whisper_stt_device – Compute device (cpu, cuda, or auto)
• --faster_whisper_stt_compute_type – Quantization type (auto, int8, float16)
• --faster_whisper_stt_gen_max_new_tokens – Token generation limit
• --faster_whisper_stt_gen_beam_size – Beam search width
• --faster_whisper_stt_gen_return_timestamps – Boolean for timestamp inclusion
• --faster_whisper_stt_gen_task – Task type (transcribe or translate)
• --faster_whisper_stt_gen_language – ISO language code (e.g., en, fr)

These flags are automatically added to the argument parser in s2s_pipeline.py using the config_prefix faster_whisper_stt.

Usage Examples

You can configure Faster Whisper via command-line arguments, Python API, or JSON configuration files.

CLI Server Mode

Deploy a realtime server with a specific Faster Whisper model on GPU:

speech-to-speech serve \
    --stt faster-whisper \
    --faster_whisper_stt_model_name large-v2 \
    --faster_whisper_stt_device cuda \
    --faster_whisper_stt_compute_type float16

Local Loopback Mode

Run both the server and audio client on the same machine using automatic device detection:

speech-to-speech local \
    --stt faster-whisper \
    --faster_whisper_stt_model_name medium.en \
    --faster_whisper_stt_device auto

Python API

Programmatically invoke the pipeline with backend-specific arguments:

from speech_to_speech.s2s_pipeline import run_pipeline_command

run_pipeline_command(
    "serve",
    [
        "--stt", "faster-whisper",
        "--faster_whisper_stt_model_name", "large-v2",
        "--faster_whisper_stt_device", "cuda",
        "--faster_whisper_stt_compute_type", "float16",
    ],
)

JSON Configuration File

Store parameters in a JSON file for reproducible deployments:

{
    "stt": "faster-whisper",
    "faster_whisper_stt_model_name": "large-v2",
    "faster_whisper_stt_device": "cuda",
    "faster_whisper_stt_compute_type": "float16"
}

Execute with:

speech-to-speech serve config.json

Summary

• The faster-whisper backend is registered in backend_registry.py with a dedicated factory function that imports FasterWhisperSTTHandler.
• Configuration is handled through FasterWhisperSTTHandlerArguments, exposing model name, device, compute type, and generation parameters.
• The handler implements setup, process, and cleanup methods to manage the Whisper model lifecycle and transcribe VAD-extracted audio chunks.
• Activation requires --stt faster-whisper followed by backend-specific flags prefixed with faster_whisper_stt_.
• Supported deployment modes include CLI server, local loopback, Python API, and JSON configuration.

Frequently Asked Questions

What Whisper models are supported when using Faster Whisper as an STT backend?

The backend accepts any model identifier supported by the underlying faster-whisper library, including tiny.en, tiny, base.en, base, small.en, small, medium.en, medium, large-v1, large-v2, and large-v3. Pass your chosen model via --faster_whisper_stt_model_name.

How do I optimize Faster Whisper for GPU inference in the speech-to-speech pipeline?

Set --faster_whisper_stt_device cuda and --faster_whisper_stt_compute_type float16 to enable half-precision GPU computation. For larger models that exceed VRAM, use int8 quantization with --faster_whisper_stt_compute_type int8 to reduce memory footprint while maintaining throughput.

Can I use Faster Whisper with CPU-only setups?

Yes. Pass --faster_whisper_stt_device cpu and select int8 quantization via --faster_whisper_stt_compute_type int8 to optimize CPU inference speed. The auto setting will also automatically detect CPU availability if no GPU is present.

Where is the actual transcription logic implemented in the source code?

The transcription logic resides in src/speech_to_speech/STT/faster_whisper_handler.py within the process method (lines 36-58). This method calls self.model.transcribe on incoming audio buffers, aggregates the resulting segments, and packages them into Transcription objects for the downstream language model handler.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}