How the Speech-to-Speech Pipeline Handles Audio Format Conversion Between Different Component Sample Rates

The Hugging Face Speech-to-Speech pipeline standardizes all internal audio to 16 kHz and automatically resamples inputs from 48 kHz WebRTC streams, 24 kHz TTS models, and arbitrary microphone rates using integer-ratio filters from scipy, torchaudio, or librosa.

The huggingface/speech-to-speech repository orchestrates multiple AI models—each with distinct native sample rates—into a real-time conversation system. Managing audio format conversion between these components is critical for seamless operation, as a mismatch between a 24 kHz text-to-speech output and a 16 kHz speech-to-text encoder would otherwise corrupt the audio stream.

The 16 kHz Pipeline Standard

All components in the S2S pipeline communicate at a single native rate defined by the constant PIPELINE_SR = 16000. This value is declared in TTS/qwen3_tts_handler.py and referenced across handlers including VAD/vad_handler.py and various TTS modules. By enforcing this standard in s2s_pipeline.py, the orchestrator ensures that voice activity detection (VAD), large language models (LLM), and speech-to-text (STT) stages can exchange tensors without manual format negotiation.

Input Stage: Resampling Incoming Audio

Microphones and WebRTC clients often provide audio at rates other than 16 kHz. The pipeline handles these conversions at entry points before the VAD or STT stages process the signal.

WebRTC Real-Time Conversion (48 kHz → 16 kHz)

Incoming Opus audio from WebRTC arrives at 48 kHz. The PcmResampler class in api/openai_realtime/webrtc_session.py uses av.AudioResampler to convert these frames to 16 kHz PCM in real-time while preserving 20 ms frame boundaries.

# api/openai_realtime/webrtc_session.py

class PcmResampler:
    """Stateless resampler that preserves state between 20 ms frames."""
    def __init__(self, target_rate: int):
        self._resampler = av.AudioResampler(format="s16", layout="mono", rate=target_rate)

    def resample_pcm(self, pcm: bytes, src_rate: int) -> bytes:
        # Decode raw PCM to an AV frame, then feed it through the resampler.

        frame = av.AudioFrame.from_ndarray(
            np.frombuffer(pcm, dtype=np.int16).reshape(-1, 1), format="s16", layout="mono"
        )
        frame.sample_rate = src_rate
        out = b"".join(r.to_ndarray().tobytes() for r in self._resampler.resample(frame))
        return out

Microphone and Arbitrary Input Handling

The VAD handler in VAD/vad_handler.py checks the incoming sample rate and lazily resamples only when necessary using torchaudio.functional.resample.

# VAD/vad_handler.py

if audio.sample_rate != self.sample_rate:
    # Convert to pipeline sample‑rate (16 kHz) using torchaudio.

    audio_float32 = torchaudio.functional.resample(
        audio, orig_freq=audio.sample_rate, new_freq=self.sample_rate
    )

Output Stage: Normalizing TTS Model Outputs

Most modern TTS models generate audio at 24 kHz, requiring downsampling to match the pipeline standard. Different handlers implement this using optimized algorithms based on the greatest common divisor (GCD) of the source and target rates.

Dynamic Integer-Ratio Resampling (Qwen-3)

The Qwen-3 handler in TTS/qwen3_tts_handler.py computes GCD-based up/down factors for scipy.signal.resample_poly to convert 24 kHz to 16 kHz efficiently.

# TTS/qwen3_tts_handler.py

from scipy.signal import resample_poly

def _resample_to_pipeline_sr(self, audio: np.ndarray, sr: int) -> np.ndarray:
    """Resample from model `sr` to the pipeline rate (16 kHz)."""
    if sr == PIPELINE_SR:
        return audio
    gcd = np.gcd(PIPELINE_SR, sr)
    return resample_poly(audio, up=PIPELINE_SR // gcd, down=sr // gcd)

Pre-Computed Resampling Factors (Pocket-TTS)

For performance-critical paths, TTS/pocket_tts_handler.py pre-calculates resampling ratios during initialization to avoid recomputing the GCD for every chunk.

# TTS/pocket_tts_handler.py

g = gcd(self.sample_rate, self.model.sample_rate)
self._resample_up = self.sample_rate // g
self._resample_down = self.model.sample_rate // g
...
chunk_resampled = resample_poly(chunk, up=self._resample_up, down=self._resample_down)

Fixed Ratio Conversion (Kokoro)

The Kokoro handler uses a simplified approach in TTS/kokoro_handler.py, calling resample_poly(up=2, down=3) to convert 24 kHz to 16 kHz without dynamic GCD calculation.

High-Quality Librosa Resampling (Facebook-MMS and ChatTTS)

For Facebook-MMS and ChatTTS models, the pipeline uses librosa.resample in facebookmms_handler.py and chatTTS_handler.py respectively, trading slightly higher CPU usage for improved audio quality.

Smart-Turn Analysis Resampling

The smart-turn analyzer in VAD/smart_turn.py handles generic audio-rate conversion for turn-detection models using scipy.signal.resample_poly with GCD-derived integer ratios. This ensures the analyzer receives correctly formatted input regardless of the source microphone's native rate.

Summary

• The pipeline enforces a strict 16 kHz standard (PIPELINE_SR) across all stages defined in the Qwen-3 handler and referenced by the VAD and orchestrator.
• Lazy resampling occurs only when sample_rate != PIPELINE_SR, avoiding unnecessary computation and CPU overhead.
• WebRTC input uses av.AudioResampler for real-time 48 kHz → 16 kHz conversion in webrtc_session.py.
• TTS outputs typically use scipy.signal.resample_poly with GCD-calculated integer ratios for efficient 24 kHz → 16 kHz downsampling in handlers like Qwen-3 and Pocket-TTS.
• Alternative implementations employ torchaudio.functional.resample for VAD input and librosa.resample for specific TTS backends requiring higher quality conversion.

Frequently Asked Questions

Why does the pipeline use 16 kHz as the standard sample rate?

16 kHz represents the native sample rate for Whisper-based STT models and provides an optimal balance between audio fidelity and computational efficiency for real-time streaming applications. Standardizing on this rate eliminates the need for repeated conversions between STT, LLM, and TTS stages.

What happens if I connect a microphone with a 44.1 kHz sample rate?

The VAD handler in VAD/vad_handler.py automatically detects the mismatch via the sample_rate attribute and resamples the audio to 16 kHz using torchaudio.functional.resample before processing begins. No manual configuration is required to support arbitrary input rates.

Does resampling introduce latency or degrade audio quality?

The pipeline employs high-quality integer-ratio polyphase resampling that minimizes distortion and phase errors. Pre-computed ratios in handlers like Pocket-TTS reduce CPU overhead during inference, while lazy conversion ensures resampling only occurs when the source rate differs from PIPELINE_SR.

Can I change the pipeline's native sample rate from 16 kHz to 24 kHz?

Changing PIPELINE_SR would require updating all component handlers—including TTS/qwen3_tts_handler.py, VAD/vad_handler.py, and s2s_pipeline.py—and verifying that downstream STT models support the new rate. The constant is hardcoded across multiple files to ensure consistency, making ad-hoc changes impractical without modifying the source code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}