How to Use Whisper as an STT Backend in the Speech-to-Speech Pipeline

To use Whisper as an STT backend, pass --stt whisper to the CLI and configure model parameters via --stt_model_name, --stt_device, and generation settings prefixed with --stt_gen_.

The huggingface/speech-to-speech repository implements a modular voice-agent pipeline (VAD → STT → LLM → TTS) where each stage is a pluggable backend. When you use Whisper as an STT backend, the system dynamically loads WhisperSTTHandler from src/speech_to_speech/STT/whisper_stt_handler.py to process audio chunks into text transcriptions.

Architecture Overview

The Speech-to-Speech pipeline uses a registry pattern to map backend names to handler implementations. When you configure Whisper as your STT provider, the system resolves the whisper key in the STT_BACKENDS registry defined in src/speech_to_speech/backend_registry.py (lines 97-108).

The data flow follows this sequence:

1. Argument Parsing – The parse_arguments function in src/speech_to_speech/s2s_pipeline.py reads the --stt flag and instantiates WhisperSTTHandlerArguments from src/speech_to_speech/arguments_classes/whisper_stt_arguments.py.

2. Configuration Normalization – The BackendSelection.normalize method converts the dataclass into a flat dictionary of kwargs, merging any generation parameters prefixed with gen_.

3. Handler Instantiation – The create_backend_handler factory dynamically imports WhisperSTTHandler and passes the normalized configuration.

4. Runtime Execution – The handler receives audio chunks from the VAD stage, preprocesses them using AutoProcessor, runs model.generate, and pushes a Transcription object downstream to the LLM.

The handler supports auto-detection of spoken languages when you specify --language auto, or you can constrain recognition to a specific language code (defined in the handler's SUPPORTED_LANGUAGES constant).

Configuration Options

Whisper-specific settings are exposed through CLI flags defined in WhisperSTTHandlerArguments. All generation parameters for the underlying transformers model use the --stt_gen_* prefix.

Key parameters:

• --stt_model_name – Hugging Face Hub model ID (e.g., openai/whisper-large-v3 or distil-whisper/distil-large-v3)
• --stt_device – Compute device (cuda, cpu, or mps)
• --stt_torch_dtype – Model precision (float16, bfloat16, or float32)
• --stt_compile_mode – Torch compile optimization mode
• --language – Language code (e.g., en, fr) or auto for auto-detection
• --stt_gen_max_new_tokens – Maximum tokens per generation
• --stt_gen_num_beams – Beam search width
• --stt_gen_temperature – Sampling temperature

Running Whisper STT

Server Mode with Whisper

Deploy the Realtime server using Whisper for speech-to-text transcription:

# Install optional Whisper dependencies

pip install "speech-to-speech[whisper]"

# Start server with Whisper large-v3 on GPU

speech-to-speech serve \
    --stt whisper \
    --stt_model_name openai/whisper-large-v3 \
    --stt_device cuda \
    --stt_torch_dtype float16 \
    --stt_gen_max_new_tokens 128 \
    --language auto

This command initializes the WhisperSTTHandler with AutoModelForSpeechSeq2Seq.from_pretrained, executes a warmup forward pass to prime CUDA graphs, and listens for WebSocket connections.

Connect a client to stream microphone audio:

speech-to-speech talk \
    --url ws://127.0.0.1:8765/v1/realtime \
    --model local

Local Mode

Run both the server and client in a single process for local testing:

speech-to-speech local \
    --stt whisper \
    --stt_model_name distil-whisper/distil-large-v3 \
    --stt_device cpu \
    --language en

Local mode instantiates the handler in the main process, bypassing the network layer while maintaining the same STT_BACKENDS registry resolution logic found in src/speech_to_speech/s2s_pipeline.py (lines 70-77).

Custom Generation Settings

Tune the transcription behavior by passing generation kwargs directly to the underlying model:

speech-to-speech serve \
    --stt whisper \
    --stt_gen_max_new_tokens 256 \
    --stt_gen_temperature 0.0 \
    --stt_gen_num_beams 4

The prepare_model_inputs method in whisper_stt_handler.py builds the gen_kwargs dictionary from these flags and passes it to model.generate().

Implementation Details

The WhisperSTTHandler class in src/speech_to_speech/STT/whisper_stt_handler.py handles model lifecycle and inference:

• Model Loading – Uses AutoProcessor.from_pretrained and AutoModelForSpeechSeq2Seq.from_pretrained with device mapping and dtype conversion
• Warmup – The warmup method runs a dummy forward pass to initialize Torch-Compile caches or CUDA graphs before processing real-time audio
• Processing – The process method implements the handler interface, receiving audio chunks and returning Transcription dataclasses

The backend registration in src/speech_to_speech/backend_registry.py wires the whisper string key to this handler class, enabling runtime selection via the --stt CLI argument.

Summary

• Use --stt whisper to select the Whisper backend when running speech-to-speech serve or speech-to-speech local
• Configure the model via --stt_model_name, --stt_device, and --stt_torch_dtype flags
• Pass generation parameters using the --stt_gen_* prefix to control beam search, temperature, and token limits
• The handler auto-detects language with --language auto or accepts specific language codes
• Source implementations reside in whisper_stt_handler.py and register via backend_registry.py

Frequently Asked Questions

What Whisper models are supported?

You can use any Whisper checkpoint available on the Hugging Face Hub, including openai/whisper-large-v3, openai/whisper-base, and distilled variants like distil-whisper/distil-large-v3. Pass the model ID to --stt_model_name.

How do I enable language auto-detection?

Add --language auto to your CLI command. The handler uses the model's built-in language classification capabilities to identify the spoken language from the audio content. Alternatively, specify a language code (e.g., --language en) to constrain recognition.

Can I use Torch compile with Whisper STT?

Yes. Pass --stt_compile_mode default or --stt_compile_mode max-autotune to enable Torch compilation. The warmup method in WhisperSTTHandler runs a dummy inference pass to compile the graph before processing real audio chunks, preventing compilation latency during live transcription.

Why is there a warmup phase when starting the server?

The warmup method executes a forward pass with dummy inputs to initialize CUDA memory pools and Torch-Compile caches. This prevents cold-start latency when the first real audio arrives from the VAD stage, ensuring consistent real-time performance for the voice pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}