Supported Audio Formats and Sample Rates in the Hugging Face Speech-to-Speech Pipeline

The Hugging Face Speech-to-Speech pipeline natively processes 16-bit PCM WAV audio at 16 kHz mono, automatically resampling any TTS output or external input to this canonical format before transmission.

The huggingface/speech-to-speech repository implements a real-time voice conversion system that standardizes all internal audio processing to a single specification. Understanding the supported audio formats and sample rates is essential for integrating custom handlers, as the pipeline enforces strict consistency across its VAD, STT, LLM, and TTS components.

Native Audio Specification

The pipeline defines its canonical audio representation in src/speech_to_speech/pipeline/messages.py, where the AudioMessage dataclass establishes the baseline for all inter-component communication.

Format and Encoding

The pipeline exclusively uses PCM-encoded WAV containers with the following characteristics:

• Container format: "wav" (defined by audio_format: str = "wav" in messages.py)
• Bit depth: 16-bit integer PCM ("s16"), as specified in src/speech_to_speech/api/openai_realtime/webrtc_session.py where the av.AudioResampler is initialized with format="s16"
• Channel layout: Mono (single channel), configured via layout="mono" in the WebRTC session resampler and enforced by setnchannels(1) in src/speech_to_speech/LLM/base_openai_compatible_language_model.py
• Sample width: 2 bytes (16 bits), set via setsampwidth(2) in the base language model's WAV writer

Sample Rate

The pipeline operates at a fixed 16 kHz (16,000 Hz) sample rate for all internal processing. This default is declared in src/speech_to_speech/pipeline/messages.py as audio_sample_rate: int = 16000, and propagated through all handler configurations.

Handling Sample Rate Mismatches

While the pipeline insists on 16 kHz for internal message passing, it accommodates components that operate at different native rates through automatic resampling and validation.

TTS Handler Resampling

Many TTS models (e.g., Pocket-TTS, Qwen-3) generate audio at 24 kHz. Rather than rejecting these outputs, TTS handlers compute the greatest common divisor (GCD) between the model's native rate and the pipeline's 16 kHz target, then apply integer up/down sampling factors.

In src/speech_to_speech/TTS/pocket_tts_handler.py, the handler calculates:

g = math.gcd(self.model.sample_rate, self.sample_rate)
self._resample_up = self.sample_rate // g
self._resample_down = self.model.sample_rate // g

This guarantees lossless conversion to the pipeline's native rate before the audio enters the message queue.

STT Input Validation

STT handlers like Whisper expect 16 kHz PCM and validate incoming audio strictly. The guard in src/speech_to_speech/VAD/smart_turn.py raises an error if sample_rate != MODEL_SAMPLE_RATE, ensuring that upstream components cannot pass incorrectly sampled data to the transcription engine.

OpenAI Realtime API Negotiation

When communicating with external APIs, the client manages format negotiation. In src/speech_to_speech/api/openai_realtime/audio_client.py, the maybe_pcm_format helper only adds a format field to the payload when the sample rate is 24 kHz; native 16 kHz PCM streams are sent without additional metadata, reducing overhead.

Practical Configuration Examples

When instantiating the pipeline, you explicitly set the sample rate to match the canonical 16 kHz standard:

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline(
    vad_handler_kwargs={"sample_rate": 16000},
    stt_handler_kwargs={"sample_rate": 16000},
    tts_handler_kwargs={"sample_rate": 16000},
)

The resulting audio messages conform to this schema:

{
    "audio_format": "wav",
    "audio_sample_rate": 16000,
    "audio": <bytes>
}

To use a TTS model with a different native rate, instantiate the handler with the pipeline rate; resampling occurs automatically:

from speech_to_speech.TTS.pocket_tts_handler import PocketTTSHandler

tts = PocketTTSHandler(sample_rate=16000)  # Target pipeline rate

# If tts.model.sample_rate is 24000, the handler resamples internally

When sending audio to the OpenAI Realtime endpoint via the AudioClient, the library handles format headers based on the source rate:

from speech_to_speech.api.openai_realtime.audio_client import AudioClient

client = AudioClient()
client.send(audio=audio_array, sample_rate=16000)  # Sent as native PCM

Summary

• The canonical format is 16-bit PCM WAV, mono, at 16 kHz, defined in src/speech_to_speech/pipeline/messages.py.
• All internal pipeline messages use this specification to ensure compatibility between VAD, STT, LLM, and TTS components.
• TTS handlers automatically resample 24 kHz model outputs to 16 kHz using GCD-based integer resampling.
• STT handlers enforce strict 16 kHz input validation and reject mismatched sample rates.
• The OpenAI Realtime API client optimizes payload metadata by omitting format fields for native 16 kHz streams.

Frequently Asked Questions

What is the default audio format used by the speech-to-speech pipeline?

The default format is PCM-encoded WAV ("wav") with 16-bit integer samples ("s16") and a 16 kHz sample rate. This is hardcoded in src/speech_to_speech/pipeline/messages.py as audio_format: str = "wav" and audio_sample_rate: int = 16000.

Why does the pipeline enforce a 16 kHz sample rate?

The 16 kHz rate provides an optimal balance between transcription accuracy for STT models (like Whisper) and computational efficiency for real-time streaming. Standardizing on a single rate eliminates format negotiation overhead between pipeline stages and ensures that the VAD, STT, and TTS handlers can operate on predictable buffer sizes.

How does the pipeline handle TTS models that output 24 kHz audio?

TTS handlers compute the greatest common divisor between the model's native sample rate (e.g., 24 kHz) and the pipeline's 16 kHz target. They then apply integer up/down sampling factors to resample the audio losslessly before wrapping it in a pipeline message, as implemented in src/speech_to_speech/TTS/pocket_tts_handler.py and qwen3_tts_handler.py.

Can I use stereo audio with the speech-to-speech pipeline?

No, the pipeline strictly requires mono (single-channel) audio. The WebRTC session resampler in src/speech_to_speech/api/openai_realtime/webrtc_session.py explicitly sets layout="mono", and the WAV writer in the base language model calls setnchannels(1). Stereo inputs must be downmixed to mono before processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}