# How the Speech-to-Speech Pipeline Handles Audio Format Conversion Between Different Component Sample Rates

> Learn how the Hugging Face Speech-to-Speech pipeline resamples audio between 16 kHz and other sample rates for seamless component integration. Discover the methods used for efficient conversion.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: internals
- Published: 2026-08-08

---

**The Hugging Face Speech-to-Speech pipeline standardizes all internal audio to 16 kHz and automatically resamples inputs from 48 kHz WebRTC streams, 24 kHz TTS models, and arbitrary microphone rates using integer-ratio filters from scipy, torchaudio, or librosa.**

The `huggingface/speech-to-speech` repository orchestrates multiple AI models—each with distinct native sample rates—into a real-time conversation system. Managing **audio format conversion** between these components is critical for seamless operation, as a mismatch between a 24 kHz text-to-speech output and a 16 kHz speech-to-text encoder would otherwise corrupt the audio stream.

## The 16 kHz Pipeline Standard

All components in the S2S pipeline communicate at a single native rate defined by the constant `PIPELINE_SR = 16000`. This value is declared in [`TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/qwen3_tts_handler.py) and referenced across handlers including [`VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/VAD/vad_handler.py) and various TTS modules. By enforcing this standard in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), the orchestrator ensures that voice activity detection (VAD), large language models (LLM), and speech-to-text (STT) stages can exchange tensors without manual format negotiation.

## Input Stage: Resampling Incoming Audio

Microphones and WebRTC clients often provide audio at rates other than 16 kHz. The pipeline handles these conversions at entry points before the VAD or STT stages process the signal.

### WebRTC Real-Time Conversion (48 kHz → 16 kHz)

Incoming Opus audio from WebRTC arrives at 48 kHz. The `PcmResampler` class in [`api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/webrtc_session.py) uses `av.AudioResampler` to convert these frames to 16 kHz PCM in real-time while preserving 20 ms frame boundaries.

```python

# api/openai_realtime/webrtc_session.py

class PcmResampler:
    """Stateless resampler that preserves state between 20 ms frames."""
    def __init__(self, target_rate: int):
        self._resampler = av.AudioResampler(format="s16", layout="mono", rate=target_rate)

    def resample_pcm(self, pcm: bytes, src_rate: int) -> bytes:
        # Decode raw PCM to an AV frame, then feed it through the resampler.

        frame = av.AudioFrame.from_ndarray(
            np.frombuffer(pcm, dtype=np.int16).reshape(-1, 1), format="s16", layout="mono"
        )
        frame.sample_rate = src_rate
        out = b"".join(r.to_ndarray().tobytes() for r in self._resampler.resample(frame))
        return out

```

### Microphone and Arbitrary Input Handling

The VAD handler in [`VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/VAD/vad_handler.py) checks the incoming sample rate and lazily resamples only when necessary using `torchaudio.functional.resample`.

```python

# VAD/vad_handler.py

if audio.sample_rate != self.sample_rate:
    # Convert to pipeline sample‑rate (16 kHz) using torchaudio.

    audio_float32 = torchaudio.functional.resample(
        audio, orig_freq=audio.sample_rate, new_freq=self.sample_rate
    )

```

## Output Stage: Normalizing TTS Model Outputs

Most modern TTS models generate audio at 24 kHz, requiring downsampling to match the pipeline standard. Different handlers implement this using optimized algorithms based on the greatest common divisor (GCD) of the source and target rates.

### Dynamic Integer-Ratio Resampling (Qwen-3)

The Qwen-3 handler in [`TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/qwen3_tts_handler.py) computes GCD-based up/down factors for `scipy.signal.resample_poly` to convert 24 kHz to 16 kHz efficiently.

```python

# TTS/qwen3_tts_handler.py

from scipy.signal import resample_poly

def _resample_to_pipeline_sr(self, audio: np.ndarray, sr: int) -> np.ndarray:
    """Resample from model `sr` to the pipeline rate (16 kHz)."""
    if sr == PIPELINE_SR:
        return audio
    gcd = np.gcd(PIPELINE_SR, sr)
    return resample_poly(audio, up=PIPELINE_SR // gcd, down=sr // gcd)

```

### Pre-Computed Resampling Factors (Pocket-TTS)

For performance-critical paths, [`TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/pocket_tts_handler.py) pre-calculates resampling ratios during initialization to avoid recomputing the GCD for every chunk.

```python

# TTS/pocket_tts_handler.py

g = gcd(self.sample_rate, self.model.sample_rate)
self._resample_up = self.sample_rate // g
self._resample_down = self.model.sample_rate // g
...
chunk_resampled = resample_poly(chunk, up=self._resample_up, down=self._resample_down)

```

### Fixed Ratio Conversion (Kokoro)

The Kokoro handler uses a simplified approach in [`TTS/kokoro_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/kokoro_handler.py), calling `resample_poly(up=2, down=3)` to convert 24 kHz to 16 kHz without dynamic GCD calculation.

### High-Quality Librosa Resampling (Facebook-MMS and ChatTTS)

For Facebook-MMS and ChatTTS models, the pipeline uses `librosa.resample` in [`facebookmms_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/facebookmms_handler.py) and [`chatTTS_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/chatTTS_handler.py) respectively, trading slightly higher CPU usage for improved audio quality.

## Smart-Turn Analysis Resampling

The smart-turn analyzer in [`VAD/smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/VAD/smart_turn.py) handles generic audio-rate conversion for turn-detection models using `scipy.signal.resample_poly` with GCD-derived integer ratios. This ensures the analyzer receives correctly formatted input regardless of the source microphone's native rate.

## Summary

- The pipeline enforces a strict **16 kHz standard** (`PIPELINE_SR`) across all stages defined in the Qwen-3 handler and referenced by the VAD and orchestrator.
- **Lazy resampling** occurs only when `sample_rate != PIPELINE_SR`, avoiding unnecessary computation and CPU overhead.
- **WebRTC input** uses `av.AudioResampler` for real-time 48 kHz → 16 kHz conversion in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py).
- **TTS outputs** typically use `scipy.signal.resample_poly` with GCD-calculated integer ratios for efficient 24 kHz → 16 kHz downsampling in handlers like Qwen-3 and Pocket-TTS.
- Alternative implementations employ `torchaudio.functional.resample` for VAD input and `librosa.resample` for specific TTS backends requiring higher quality conversion.

## Frequently Asked Questions

### Why does the pipeline use 16 kHz as the standard sample rate?

16 kHz represents the native sample rate for Whisper-based STT models and provides an optimal balance between audio fidelity and computational efficiency for real-time streaming applications. Standardizing on this rate eliminates the need for repeated conversions between STT, LLM, and TTS stages.

### What happens if I connect a microphone with a 44.1 kHz sample rate?

The VAD handler in [`VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/VAD/vad_handler.py) automatically detects the mismatch via the `sample_rate` attribute and resamples the audio to 16 kHz using `torchaudio.functional.resample` before processing begins. No manual configuration is required to support arbitrary input rates.

### Does resampling introduce latency or degrade audio quality?

The pipeline employs high-quality **integer-ratio polyphase resampling** that minimizes distortion and phase errors. Pre-computed ratios in handlers like Pocket-TTS reduce CPU overhead during inference, while lazy conversion ensures resampling only occurs when the source rate differs from `PIPELINE_SR`.

### Can I change the pipeline's native sample rate from 16 kHz to 24 kHz?

Changing `PIPELINE_SR` would require updating all component handlers—including [`TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/TTS/qwen3_tts_handler.py), [`VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/VAD/vad_handler.py), and [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)—and verifying that downstream STT models support the new rate. The constant is hardcoded across multiple files to ensure consistency, making ad-hoc changes impractical without modifying the source code.