How to Manage Audio Format Conversions and Resampling in the Hugging Face Speech-to-Speech Pipeline

The Speech-to-Speech pipeline standardizes all audio to a single 16 kHz sample rate using stateful resamplers for real-time streams and poly-phase filtering for model-generated audio.

All audio entering or leaving the system passes through dedicated conversion layers. Whether you're ingesting WebRTC Opus streams, processing TTS outputs from models like Pocket-TTS or Kokoro, or sending audio back to clients, the pipeline enforces a consistent internal format. This guide explains how these conversions work and how to implement them in your own extensions.

The Pipeline's Central Audio Contract

The Speech-to-Speech repository operates on a strict 16 kHz mono PCM standard. Every component—from voice activity detection (VAD) to speech-to-text (STT) to text-to-speech (TTS)—expects this format. This design eliminates sample rate mismatches between modular handlers.

As implemented in huggingface/speech-to-speech, three conversion layers bridge external audio to this internal standard:

• Inbound resampling: Client audio (typically 48 kHz Opus) → 16 kHz PCM
• Internal normalization: Model-specific rates (24 kHz, 44.1 kHz, etc.) → 16 kHz PCM
• Outbound resampling: 16 kHz PCM → Client-desired rate (usually 48 kHz)

Stateful Resampling for Real-Time Streams

The WebRTC session handler manages bidirectional audio streaming with minimal latency. It uses a stateful PcmResampler class that preserves filter state across successive 20 ms frames, preventing audible glitches at frame boundaries.

How PcmResampler Works

Located in src/speech_to_speech/api/openai_realtime/webrtc_session.py, this class wraps FFmpeg's av.AudioResampler:

from speech_to_speech.api.openai_realtime.webrtc_session import PcmResampler

# Incoming: 48 kHz WebRTC → 16 kHz pipeline

inbound_resampler = PcmResampler(target_rate=16_000)
pcm_16k = inbound_resampler.resample_pcm(opus_decoded_pcm, src_rate=48_000)

# Outgoing: 16 kHz pipeline → 48 kHz WebRTC

outbound_resampler = PcmResampler(target_rate=48_000)
pcm_48k = outbound_resampler.resample_pcm(pipeline_pcm, src_rate=16_000)

The resampler also handles channel down-mixing (stereo to mono), ensuring the pipeline receives the expected single-channel format.

Resampling Model-Generated Audio

Different TTS backends operate at their native sample rates. Each handler converts output to 16 kHz before returning audio to the pipeline.

Pocket-TTS: Poly-Phase Resampling with SciPy

The Pocket-TTS handler in pocket_tts_handler.py uses scipy.signal.resample_poly for efficient poly-phase filtering:

from scipy.signal import resample_poly
import numpy as np

def resample_to_pipeline(waveform: np.ndarray, model_sr: int = 24_000) -> np.ndarray:
    """Convert Pocket-TTS 24 kHz output to pipeline 16 kHz."""
    target_sr = 16_000
    
    # Compute integer up/down factors for exact rational resampling

    g = np.gcd(model_sr, target_sr)
    up, down = target_sr // g, model_sr // g
    
    resampled = resample_poly(waveform, up=up, down=down)
    return (resampled * 32768).astype(np.int16)

The np.gcd calculation ensures rational resampling factors, which resample_poly implements efficiently without successive interpolation/decimation stages.

Other TTS Handlers: Library-Specific Approaches

Handler	Source Rate	Method	File
Kokoro	24 kHz	resample_poly	kokoro_handler.py
Qwen-3	Variable	resample_poly	qwen3_tts_handler.py
Facebook-MMS	Varies	librosa.resample	facebookmms_handler.py
ChatTTS	24 kHz	torchaudio.functional.resample	chatTTS_handler.py

All implementations converge on the same 16 kHz np.int16 output format.

VAD and Audio Enhancement Resampling

The VAD handler optionally resamples for deep-filter-based enhancement. In vad_handler.py, Torchaudio provides GPU-accelerated resampling when available:

import torchaudio
import torch

def enhance_resample(array: np.ndarray, sample_rate: int, target_sr: int) -> torch.Tensor:
    """Resample for enhancement model processing."""
    return torchaudio.functional.resample(
        torch.from_numpy(array),
        orig_freq=sample_rate,
        new_freq=target_sr,  # Typically matches df_state.sr()

    )

The enhancement model's expected rate (self.df_state.sr()) may differ from the pipeline rate, requiring on-the-fly conversion.

Utility Functions for Generic Conversion

For ad-hoc resampling needs, utils.py provides a convenience wrapper around the same poly-phase logic:

from speech_to_speech.api.openai_realtime.utils import resample

# Convert arbitrary-rate PCM bytes to pipeline format

pipeline_pcm = resample(
    pcm_bytes,           # Raw int16 bytes

    from_rate=44_100,    # Source sample rate

    to_rate=16_000       # Target pipeline rate

)

This function handles byte-to-array conversion, resampling, and format normalization in one call.

Implementing Custom Audio Handlers

When adding a new TTS backend or audio source, follow this pattern:

1. Accept the native sample rate from your model or source
2. Compute GCD-based integer factors for rational resampling when using resample_poly
3. Output int16 at 16 kHz to maintain pipeline compatibility
4. Use stateful resamplers for streaming contexts to avoid boundary artifacts

Example: Custom TTS Handler Template

import numpy as np
from scipy.signal import resample_poly

class CustomTTSHandler:
    def __init__(self, sample_rate: int = 16_000):
        self.sample_rate = sample_rate  # Pipeline rate

    
    def process(self, text: str) -> np.ndarray:
        # Generate audio at model's native rate

        waveform, model_sr = self.tts_model.synthesize(text)
        
        # Normalize to pipeline rate

        if model_sr != self.sample_rate:
            g = np.gcd(model_sr, self.sample_rate)
            up, down = self.sample_rate // g, model_sr // g
            waveform = resample_poly(waveform, up=up, down=down)
        
        # Ensure int16 format

        return (waveform * 32767).clip(-32768, 32767).astype(np.int16)

Key Files and Their Responsibilities

File	Purpose
webrtc_session.py	Stateful PcmResampler for WebRTC bidirectional streaming
utils.py	Generic resample() helper for arbitrary rate conversion
pocket_tts_handler.py	Reference implementation of resample_poly for TTS
qwen3_tts_handler.py	Alternative TTS with rational resampling
kokoro_handler.py	24 kHz → 16 kHz conversion
facebookmms_handler.py	Librosa-based resampling example
chatTTS_handler.py	TorchAudio functional resampling
vad_handler.py	Enhancement-time resampling with Torchaudio
smart_turn.py	Fallback resampling for turn detection

Summary

• Single internal format: The pipeline enforces 16 kHz mono PCM throughout all processing stages
• Stateful streaming: PcmResampler in webrtc_session.py maintains filter state for glitch-free real-time conversion
• Model normalization: TTS handlers use resample_poly, librosa.resample, or torchaudio.functional.resample to reach 16 kHz
• Rational factors: GCD-based integer up/down ratios preserve audio quality in SciPy-based resampling
• Bidirectional symmetry: The same resampler class handles both inbound (client→pipeline) and outbound (pipeline→client) conversion

Frequently Asked Questions

What sample rate does the Speech-to-Speech pipeline use internally?

The pipeline operates at 16 kHz mono 16-bit PCM. All components—from VAD to STT to TTS—exchange audio in this format, regardless of external source or destination rates.

Why use resample_poly instead of simple interpolation?

resample_poly implements poly-phase filtering with exact rational resampling factors (up/down integers). This avoids successive upsampling/downsampling stages, reduces aliasing, and runs faster than general-purpose resamplers while maintaining quality suitable for speech.

How does the pipeline prevent audio glitches in streaming scenarios?

The PcmResampler class maintains filter state across frames, so successive 20 ms chunks are processed with continuous phase. Without state preservation, boundary discontinuities would create audible clicks or pops in real-time streams.

Can I use the built-in resampling utilities outside of the WebRTC context?

Yes. Import resample from utils.py or instantiate PcmResampler directly for any PCM conversion need. These utilities work independently of the WebRTC session infrastructure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}