How to Handle Audio Format Conversion Between Pipeline Stages in Speech-to-Speech

The Speech-to-Speech framework handles audio format conversion between pipeline stages through five core transformations: PCM bytes to float32 NumPy arrays, NumPy to in-memory WAV containers, WAV files back to NumPy, sample-rate resampling, and final byte encoding for transport—each implemented in specific handler modules to maintain a unified 16 kHz pipeline.

Audio format conversion is the invisible glue that holds real-time speech-to-speech pipelines together. In the huggingface/speech-to-speech repository, microphone input, speech-to-text (STT) models, language model processing, and text-to-speech (TTS) generation all speak slightly different audio dialects. The framework resolves these mismatches through systematic conversions implemented across VAD/, LLM/, and TTS/ handler modules, ensuring seamless data flow from raw PCM bytes to synthesized speech output.

Understanding Pipeline Stage Requirements

Each pipeline stage in Speech-to-Speech expects and produces audio in distinct formats. The microphone and voice activity detection (VAD) front-end operates on raw PCM-16 mono bytes at 16 kHz. STT handlers consume float32 NumPy arrays directly. The LLM processing layer requires base64-encoded WAV data URLs for audio understanding. TTS back-ends accept either WAV files or NumPy arrays and return waveform data that ultimately streams back to the client as raw bytes.

This format diversity demands precise conversion utilities. Rather than forcing all components into a single representation, the codebase strategically transforms audio at stage boundaries—preserving model-native formats where efficient and converting only where necessary.

Conversion 1: PCM Bytes to Float32 NumPy Arrays (VAD Stage)

The first conversion occurs at the pipeline's entry point. The LocalAudioStreamer captures 16-bit PCM mono bytes at 16 kHz, but VAD and STT processing requires normalized float32 NumPy arrays.

In VAD/vad_iterator.py, raw bytes flow through a standardized transformation pipeline:

# From speech_to_speech/utils/utils.py

import numpy as np

def int2float(sound):
    """Convert 16-bit PCM integers to float32 range [-1.0, 1.0]"""
    sound = sound.astype('float32')
    sound *= 1 / 32768
    return sound.squeeze()

The VAD iterator reads PCM frames, assembles them into NumPy int16 arrays via np.frombuffer(), then applies int2float() to normalize to the ±1.0 range expected by Silero VAD models and downstream STT handlers.

Key implementation details:

• Source: src/speech_to_speech/utils/utils.py (lines 46+)
• Used by: src/speech_to_speech/VAD/vad_iterator.py
• Output shape: (num_samples,) float32 array
• Normalization factor: 1/32768 (16-bit signed integer range)

Conversion 2: NumPy Arrays to In-Memory WAV (LLM Stage)

When the pipeline routes audio to language models for spoken dialogue understanding, it must package NumPy waveforms as WAV files—without disk I/O. The _audio_to_wav_base64() helper in LLM/base_openai_compatible_language_model.py constructs in-memory WAV containers using Python's standard library wave module:

# Condensed from base_openai_compatible_language_model.py

import wave
import base64
import io

def _audio_to_wav_base64(audio_array, sample_rate=16000):
    """Convert float32 NumPy array to base64-encoded WAV data URL."""
    # Ensure int16 range

    audio_int16 = (audio_array * 32768).astype(np.int16)
    
    buffer = io.BytesIO()
    with wave.open(buffer, 'wb') as wav_file:
        wav_file.setnchannels(1)      # Mono

        wav_file.setsampwidth(2)      # 16-bit

        wav_file.setframerate(sample_rate)
        wav_file.writeframes(audio_int16.tobytes())
    
    wav_bytes = base64.b64encode(buffer.getvalue()).decode('utf-8')
    return f"data:audio/wav;base64,{wav_bytes}"

This conversion enables OpenAI-compatible multimodal APIs to receive reference audio. The data URL format (data:audio/wav;base64,...) embeds directly in JSON payloads without external file references.

Critical parameters:

• Channel count: 1 (mono)
• Sample width: 2 bytes (16-bit PCM)
• Frame rate: Pipeline-standard 16 kHz
• Encoding: Base64 with UTF-8 string output

Conversion 3: WAV Files to NumPy Arrays (TTS Reference Audio)

TTS handlers accepting reference audio for voice cloning must load WAV files and prepare them for model inference. The Qwen-3 TTS handler implements this in load_wav_and_resample():

# From qwen3_tts_handler.py

import soundfile as sf
from scipy.signal import resample_poly

def load_wav_and_resample(file_path, target_sr=16000):
    waveform, orig_sr = sf.read(file_path, dtype="float32")
    
    # Force mono: average channels if stereo

    if waveform.ndim > 1:
        waveform = waveform.mean(axis=1)
    
    # Resample to pipeline standard if needed

    if orig_sr != target_sr:
        waveform = resample_poly(waveform, up=target_sr, down=orig_sr)
    
    return waveform

Implementation notes:

• Source: src/speech_to_speech/TTS/qwen3_tts_handler.py (lines 464+)
• Library: soundfile (sf.read) for robust WAV/FLAC/OGG support
• Resampling: scipy.signal.resample_poly for rational-factor resampling
• Target: Consistent 16 kHz mono float32 for Qwen-3 tokenization

Conversion 4: Sample-Rate Resampling Between Model Native Rates

Different TTS models train at different sample rates. The pipeline standardizes on 16 kHz for transport and VAD compatibility, requiring on-the-fly resampling.

Qwen-3 Handler: scipy.signal.resample_poly

Qwen-3 uses resample_poly for high-quality rational resampling when reference audio differs from the target rate:

waveform = resample_poly(waveform, up=16000, down=original_sr)

This method applies polyphase filtering, avoiding aliasing while preserving phase coherence—critical for voice cloning accuracy.

ChatTTS Handler: librosa.resample

ChatTTS generates audio at 24 kHz but must output 16 kHz for client playback. In chatTTS_handler.py (lines 85+), the handler applies:

import librosa

audio_16k = librosa.resample(
    audio_chattts, 
    orig_sr=24000, 
    target_sr=16000,
    res_type='kaiser_best'  # High-quality mode

)

Resampling strategy comparison:

Handler	Library	Method	Use Case
Qwen-3	SciPy	resample_poly	Rational-factor resampling of reference audio
ChatTTS	Librosa	resample with Kaiser window	Arbitrary-rate conversion of generated output
Facebook MMS	None	Direct tensor extraction	Model outputs at target rate natively

Conversion 5: NumPy to Transport Bytes (Client Streaming)

The final conversion prepares synthesized audio for WebSocket transmission. TTS handlers return float32 NumPy waveforms, which must become transmittable bytes:

# Final encoding before WebSocket send

pcm_bytes = (generated_audio * 32767).astype(np.int16).tobytes()

The client receives raw PCM or re-wraps it as a WAV data URL for browser Audio element playback. The demo script scripts/synthetic_conversation_realtime_client.py demonstrates this client-side decoding.

Complete Conversion Pipeline Walkthrough

Tracing a single utterance through the system illustrates how these conversions chain together:

1. Microphone → VAD: b'\x00\x01\xff\xfe...' (PCM-16) → int2float() → np.ndarray(float32, 16kHz)
2. VAD → STT: NumPy array passes directly (no conversion)
3. STT → LLM (optional audio): _audio_to_wav_base64() → data:audio/wav;base64,...
4. LLM → TTS: Text (or decoded audio) → synthesis parameters
5. TTS generation: Model-native rate (e.g., 24 kHz) → librosa.resample() or direct output → 16 kHz NumPy
6. TTS → Client: astype(np.int16).tobytes() → WebSocket binary → browser playback

Why These Conversions Matter

Interoperability: The unified 16 kHz pipeline allows mixing and matching VAD, STT, LLM, and TTS components from different vendors without format mismatches.

Efficiency: In-memory WAV generation avoids filesystem overhead. Rational resampling (resample_poly) outperforms naive interpolation for common rate ratios like 48→16 kHz.

Extensibility: New TTS back-ends only need to implement generate(audio_array) -> np.ndarray at 16 kHz. The surrounding scaffolding in BaseHandler classes manages all upstream and downstream conversions automatically.

Summary

• PCM to NumPy: utils.int2float() in utils/utils.py normalizes 16-bit integers to float32 range
• NumPy to WAV: _audio_to_wav_base64() in LLM/base_openai_compatible_language_model.py creates data URLs for LLM APIs
• WAV to NumPy: soundfile.read() in TTS handlers loads reference audio with automatic format detection
• Resampling: scipy.signal.resample_poly (Qwen-3) and librosa.resample (ChatTTS) handle rate mismatches
• Final encoding: Int16 byte conversion prepares audio for WebSocket streaming to clients

Frequently Asked Questions

What sample rate does the Speech-to-Speech pipeline use internally?

The pipeline standardizes on 16 kHz mono audio for all internal transport and VAD processing. Individual models may train or generate at different rates (e.g., ChatTTS at 24 kHz), but handlers resample to 16 kHz before returning waveforms to the pipeline. This consistency ensures microphone input, STT recognition, and client playback all share compatible formats.

Why base64-encode WAV data instead of sending raw NumPy arrays to LLMs?

Base64-encoded WAV data URLs embed directly in JSON payloads required by OpenAI-compatible chat completion APIs. Raw NumPy arrays lack standard container metadata (sample rate, channel count, bit depth) that multimodal models need for proper decoding. The _audio_to_wav_base64() helper preserves this metadata in a universally parseable format without filesystem dependencies.

How does the pipeline handle stereo audio inputs?

TTS reference audio loaders force mono conversion by averaging stereo channels: waveform.mean(axis=1) in qwen3_tts_handler.py. The microphone front-end captures mono exclusively. All downstream processing assumes single-channel audio to reduce bandwidth and match training distributions of most speech models.

Can I use a different target sample rate than 16 kHz?

Changing the pipeline rate requires coordinated modifications across multiple handlers. The 16 kHz assumption is hardcoded in: utils.int2float normalization constants, VADIterator frame expectations, _audio_to_wav_base64 header settings, and several TTS resampling conditionals. For custom rates, override these utilities consistently or implement a sample_rate configuration propagated through BaseHandler initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}