# Supported Audio Formats and Sample Rates in the Hugging Face Speech-to-Speech Pipeline

> Discover the audio formats and sample rates supported by the Hugging Face Speech-to-Speech pipeline. Learn about native processing and automatic resampling for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: api-reference
- Published: 2026-08-10

---

**The Hugging Face Speech-to-Speech pipeline natively processes 16-bit PCM WAV audio at 16 kHz mono, automatically resampling any TTS output or external input to this canonical format before transmission.**

The `huggingface/speech-to-speech` repository implements a real-time voice conversion system that standardizes all internal audio processing to a single specification. Understanding the supported audio formats and sample rates is essential for integrating custom handlers, as the pipeline enforces strict consistency across its VAD, STT, LLM, and TTS components.

## Native Audio Specification

The pipeline defines its canonical audio representation in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py), where the `AudioMessage` dataclass establishes the baseline for all inter-component communication.

### Format and Encoding

The pipeline exclusively uses **PCM-encoded WAV** containers with the following characteristics:

- **Container format**: `"wav"` (defined by `audio_format: str = "wav"` in [`messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/messages.py))
- **Bit depth**: 16-bit integer PCM (`"s16"`), as specified in [`src/speech_to_speech/api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py) where the `av.AudioResampler` is initialized with `format="s16"`
- **Channel layout**: Mono (single channel), configured via `layout="mono"` in the WebRTC session resampler and enforced by `setnchannels(1)` in [`src/speech_to_speech/LLM/base_openai_compatible_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/base_openai_compatible_language_model.py)
- **Sample width**: 2 bytes (16 bits), set via `setsampwidth(2)` in the base language model's WAV writer

### Sample Rate

The pipeline operates at a fixed **16 kHz** (16,000 Hz) sample rate for all internal processing. This default is declared in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) as `audio_sample_rate: int = 16000`, and propagated through all handler configurations.

## Handling Sample Rate Mismatches

While the pipeline insists on 16 kHz for internal message passing, it accommodates components that operate at different native rates through automatic resampling and validation.

### TTS Handler Resampling

Many TTS models (e.g., Pocket-TTS, Qwen-3) generate audio at 24 kHz. Rather than rejecting these outputs, TTS handlers compute the greatest common divisor (GCD) between the model's native rate and the pipeline's 16 kHz target, then apply integer up/down sampling factors.

In [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py), the handler calculates:

```python
g = math.gcd(self.model.sample_rate, self.sample_rate)
self._resample_up = self.sample_rate // g
self._resample_down = self.model.sample_rate // g

```

This guarantees lossless conversion to the pipeline's native rate before the audio enters the message queue.

### STT Input Validation

STT handlers like Whisper expect 16 kHz PCM and validate incoming audio strictly. The guard in [`src/speech_to_speech/VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/smart_turn.py) raises an error if `sample_rate != MODEL_SAMPLE_RATE`, ensuring that upstream components cannot pass incorrectly sampled data to the transcription engine.

### OpenAI Realtime API Negotiation

When communicating with external APIs, the client manages format negotiation. In [`src/speech_to_speech/api/openai_realtime/audio_client.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/audio_client.py), the `maybe_pcm_format` helper only adds a `format` field to the payload when the sample rate is 24 kHz; native 16 kHz PCM streams are sent without additional metadata, reducing overhead.

## Practical Configuration Examples

When instantiating the pipeline, you explicitly set the sample rate to match the canonical 16 kHz standard:

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline(
    vad_handler_kwargs={"sample_rate": 16000},
    stt_handler_kwargs={"sample_rate": 16000},
    tts_handler_kwargs={"sample_rate": 16000},
)

```

The resulting audio messages conform to this schema:

```python
{
    "audio_format": "wav",
    "audio_sample_rate": 16000,
    "audio": <bytes>
}

```

To use a TTS model with a different native rate, instantiate the handler with the pipeline rate; resampling occurs automatically:

```python
from speech_to_speech.TTS.pocket_tts_handler import PocketTTSHandler

tts = PocketTTSHandler(sample_rate=16000)  # Target pipeline rate

# If tts.model.sample_rate is 24000, the handler resamples internally

```

When sending audio to the OpenAI Realtime endpoint via the `AudioClient`, the library handles format headers based on the source rate:

```python
from speech_to_speech.api.openai_realtime.audio_client import AudioClient

client = AudioClient()
client.send(audio=audio_array, sample_rate=16000)  # Sent as native PCM

```

## Summary

- The **canonical format** is 16-bit PCM WAV, mono, at **16 kHz**, defined in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py).
- All internal pipeline messages use this specification to ensure compatibility between VAD, STT, LLM, and TTS components.
- **TTS handlers** automatically resample 24 kHz model outputs to 16 kHz using GCD-based integer resampling.
- **STT handlers** enforce strict 16 kHz input validation and reject mismatched sample rates.
- The **OpenAI Realtime API client** optimizes payload metadata by omitting format fields for native 16 kHz streams.

## Frequently Asked Questions

### What is the default audio format used by the speech-to-speech pipeline?

The default format is **PCM-encoded WAV** (`"wav"`) with **16-bit integer samples** (`"s16"`) and a **16 kHz** sample rate. This is hardcoded in [`src/speech_to_speech/pipeline/messages.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/messages.py) as `audio_format: str = "wav"` and `audio_sample_rate: int = 16000`.

### Why does the pipeline enforce a 16 kHz sample rate?

The 16 kHz rate provides an optimal balance between transcription accuracy for STT models (like Whisper) and computational efficiency for real-time streaming. Standardizing on a single rate eliminates format negotiation overhead between pipeline stages and ensures that the VAD, STT, and TTS handlers can operate on predictable buffer sizes.

### How does the pipeline handle TTS models that output 24 kHz audio?

TTS handlers compute the greatest common divisor between the model's native sample rate (e.g., 24 kHz) and the pipeline's 16 kHz target. They then apply integer up/down sampling factors to resample the audio losslessly before wrapping it in a pipeline message, as implemented in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) and [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py).

### Can I use stereo audio with the speech-to-speech pipeline?

No, the pipeline strictly requires **mono** (single-channel) audio. The WebRTC session resampler in [`src/speech_to_speech/api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py) explicitly sets `layout="mono"`, and the WAV writer in the base language model calls `setnchannels(1)`. Stereo inputs must be downmixed to mono before processing.