# How to Manage Audio Format Conversions and Resampling in the Hugging Face Speech-to-Speech Pipeline

> Learn to manage audio format conversions and resampling in the Hugging Face Speech-to-Speech pipeline. Discover how it handles real-time streams and model-generated audio efficiently.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**The Speech-to-Speech pipeline standardizes all audio to a single 16 kHz sample rate using stateful resamplers for real-time streams and poly-phase filtering for model-generated audio.**

All audio entering or leaving the system passes through dedicated conversion layers. Whether you're ingesting WebRTC Opus streams, processing TTS outputs from models like Pocket-TTS or Kokoro, or sending audio back to clients, the pipeline enforces a consistent internal format. This guide explains how these conversions work and how to implement them in your own extensions.

## The Pipeline's Central Audio Contract

The Speech-to-Speech repository operates on a strict **16 kHz mono PCM** standard. Every component—from voice activity detection (VAD) to speech-to-text (STT) to text-to-speech (TTS)—expects this format. This design eliminates sample rate mismatches between modular handlers.

As implemented in `huggingface/speech-to-speech`, three conversion layers bridge external audio to this internal standard:

- **Inbound resampling**: Client audio (typically 48 kHz Opus) → 16 kHz PCM
- **Internal normalization**: Model-specific rates (24 kHz, 44.1 kHz, etc.) → 16 kHz PCM
- **Outbound resampling**: 16 kHz PCM → Client-desired rate (usually 48 kHz)

## Stateful Resampling for Real-Time Streams

The **WebRTC session handler** manages bidirectional audio streaming with minimal latency. It uses a stateful `PcmResampler` class that preserves filter state across successive 20 ms frames, preventing audible glitches at frame boundaries.

### How PcmResampler Works

Located in [`src/speech_to_speech/api/openai_realtime/webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/webrtc_session.py), this class wraps FFmpeg's `av.AudioResampler`:

```python
from speech_to_speech.api.openai_realtime.webrtc_session import PcmResampler

# Incoming: 48 kHz WebRTC → 16 kHz pipeline

inbound_resampler = PcmResampler(target_rate=16_000)
pcm_16k = inbound_resampler.resample_pcm(opus_decoded_pcm, src_rate=48_000)

# Outgoing: 16 kHz pipeline → 48 kHz WebRTC

outbound_resampler = PcmResampler(target_rate=48_000)
pcm_48k = outbound_resampler.resample_pcm(pipeline_pcm, src_rate=16_000)

```

The resampler also handles **channel down-mixing** (stereo to mono), ensuring the pipeline receives the expected single-channel format.

## Resampling Model-Generated Audio

Different TTS backends operate at their native sample rates. Each handler converts output to 16 kHz before returning audio to the pipeline.

### Pocket-TTS: Poly-Phase Resampling with SciPy

The Pocket-TTS handler in [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) uses `scipy.signal.resample_poly` for efficient poly-phase filtering:

```python
from scipy.signal import resample_poly
import numpy as np

def resample_to_pipeline(waveform: np.ndarray, model_sr: int = 24_000) -> np.ndarray:
    """Convert Pocket-TTS 24 kHz output to pipeline 16 kHz."""
    target_sr = 16_000
    
    # Compute integer up/down factors for exact rational resampling

    g = np.gcd(model_sr, target_sr)
    up, down = target_sr // g, model_sr // g
    
    resampled = resample_poly(waveform, up=up, down=down)
    return (resampled * 32768).astype(np.int16)

```

The `np.gcd` calculation ensures **rational resampling factors**, which `resample_poly` implements efficiently without successive interpolation/decimation stages.

### Other TTS Handlers: Library-Specific Approaches

| Handler | Source Rate | Method | File |
|---------|-------------|--------|------|
| Kokoro | 24 kHz | `resample_poly` | [`kokoro_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/kokoro_handler.py) |
| Qwen-3 | Variable | `resample_poly` | [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) |
| Facebook-MMS | Varies | `librosa.resample` | [`facebookmms_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/facebookmms_handler.py) |
| ChatTTS | 24 kHz | `torchaudio.functional.resample` | [`chatTTS_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/chatTTS_handler.py) |

All implementations converge on the same 16 kHz `np.int16` output format.

## VAD and Audio Enhancement Resampling

The VAD handler optionally resamples for deep-filter-based enhancement. In [`vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_handler.py), Torchaudio provides GPU-accelerated resampling when available:

```python
import torchaudio
import torch

def enhance_resample(array: np.ndarray, sample_rate: int, target_sr: int) -> torch.Tensor:
    """Resample for enhancement model processing."""
    return torchaudio.functional.resample(
        torch.from_numpy(array),
        orig_freq=sample_rate,
        new_freq=target_sr,  # Typically matches df_state.sr()

    )

```

The enhancement model's expected rate (`self.df_state.sr()`) may differ from the pipeline rate, requiring on-the-fly conversion.

## Utility Functions for Generic Conversion

For ad-hoc resampling needs, [`utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils.py) provides a convenience wrapper around the same poly-phase logic:

```python
from speech_to_speech.api.openai_realtime.utils import resample

# Convert arbitrary-rate PCM bytes to pipeline format

pipeline_pcm = resample(
    pcm_bytes,           # Raw int16 bytes

    from_rate=44_100,    # Source sample rate

    to_rate=16_000       # Target pipeline rate

)

```

This function handles byte-to-array conversion, resampling, and format normalization in one call.

## Implementing Custom Audio Handlers

When adding a new TTS backend or audio source, follow this pattern:

1. **Accept the native sample rate** from your model or source
2. **Compute GCD-based integer factors** for rational resampling when using `resample_poly`
3. **Output int16 at 16 kHz** to maintain pipeline compatibility
4. **Use stateful resamplers** for streaming contexts to avoid boundary artifacts

### Example: Custom TTS Handler Template

```python
import numpy as np
from scipy.signal import resample_poly

class CustomTTSHandler:
    def __init__(self, sample_rate: int = 16_000):
        self.sample_rate = sample_rate  # Pipeline rate

    
    def process(self, text: str) -> np.ndarray:
        # Generate audio at model's native rate

        waveform, model_sr = self.tts_model.synthesize(text)
        
        # Normalize to pipeline rate

        if model_sr != self.sample_rate:
            g = np.gcd(model_sr, self.sample_rate)
            up, down = self.sample_rate // g, model_sr // g
            waveform = resample_poly(waveform, up=up, down=down)
        
        # Ensure int16 format

        return (waveform * 32767).clip(-32768, 32767).astype(np.int16)

```

## Key Files and Their Responsibilities

| File | Purpose |
|------|---------|
| [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) | Stateful `PcmResampler` for WebRTC bidirectional streaming |
| [`utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils.py) | Generic `resample()` helper for arbitrary rate conversion |
| [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) | Reference implementation of `resample_poly` for TTS |
| [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) | Alternative TTS with rational resampling |
| [`kokoro_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/kokoro_handler.py) | 24 kHz → 16 kHz conversion |
| [`facebookmms_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/facebookmms_handler.py) | Librosa-based resampling example |
| [`chatTTS_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/chatTTS_handler.py) | TorchAudio functional resampling |
| [`vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_handler.py) | Enhancement-time resampling with Torchaudio |
| [`smart_turn.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_turn.py) | Fallback resampling for turn detection |

## Summary

- **Single internal format**: The pipeline enforces 16 kHz mono PCM throughout all processing stages
- **Stateful streaming**: `PcmResampler` in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) maintains filter state for glitch-free real-time conversion
- **Model normalization**: TTS handlers use `resample_poly`, `librosa.resample`, or `torchaudio.functional.resample` to reach 16 kHz
- **Rational factors**: GCD-based integer up/down ratios preserve audio quality in SciPy-based resampling
- **Bidirectional symmetry**: The same resampler class handles both inbound (client→pipeline) and outbound (pipeline→client) conversion

## Frequently Asked Questions

### What sample rate does the Speech-to-Speech pipeline use internally?

The pipeline operates at **16 kHz mono 16-bit PCM**. All components—from VAD to STT to TTS—exchange audio in this format, regardless of external source or destination rates.

### Why use `resample_poly` instead of simple interpolation?

`resample_poly` implements **poly-phase filtering** with exact rational resampling factors (up/down integers). This avoids successive upsampling/downsampling stages, reduces aliasing, and runs faster than general-purpose resamplers while maintaining quality suitable for speech.

### How does the pipeline prevent audio glitches in streaming scenarios?

The `PcmResampler` class maintains **filter state across frames**, so successive 20 ms chunks are processed with continuous phase. Without state preservation, boundary discontinuities would create audible clicks or pops in real-time streams.

### Can I use the built-in resampling utilities outside of the WebRTC context?

Yes. Import `resample` from [`utils.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils.py) or instantiate `PcmResampler` directly for any PCM conversion need. These utilities work independently of the WebRTC session infrastructure.