How to Debug Audio Pipeline Latency and Dropped Frames in Speech-to-Speech: A Complete Guide

Set LOG_LEVEL=DEBUG, monitor queue sizes in s2s_pipeline.py, and inspect _log_first_audio_latency in TTS handlers to identify whether latency stems from capture blocksize, model inference, network jitter, or queue back-pressure.

The Hugging Face speech-to-speech repository implements a real-time audio pipeline connecting microphone capture, voice-activity detection, speech-to-text inference, large language model generation, and text-to-speech playback. Debugging audio pipeline latency and dropped frames requires tracing timing across six interconnected stages, each with distinct failure modes and diagnostic hooks.

Understanding the Pipeline Architecture

The system processes audio through a chain of queues and background threads orchestrated by s2s_pipeline.py. Latency and frame loss manifest differently depending on which stage becomes the bottleneck.

Stage	Core Module	Latency Risk
Capture / Playback	LocalAudioStreamer (local_audio_streamer.py)	blocksize parameter directly sets capture-to-playback delay
WebSocket Transport	WebsocketStreamer (websocket_streamer.py)	Network jitter causes frame misalignment beyond AUDIO_PTIME = 0.02
Resampling	WebrtcAudioTrack (webrtc_session.py)	48 kHz → 16 kHz conversion adds buffering delay
Voice-Activity Detection	VADIterator (vad_iterator.py)	Queue back-pressure when processing lags behind capture
STT Inference	Handlers like MLXAudioWhisperHandler	Model inference stalls downstream components
TTS Inference	Handlers like Qwen3TTSHandler	First-frame latency from model warm-up or GPU graph capture

Diagnosing Capture and Playback Issues

Blocksize Configuration in LocalAudioStreamer

The LocalAudioStreamer class in src/speech_to_speech/connections/local_audio_streamer.py uses sounddevice.Stream with a default blocksize=512 samples. At 16 kHz sampling rate, this equals approximately 32 ms of audio per callback.

Too large: Increases end-to-end latency.
Too small: Raises CPU utilization and risk of dropouts.

Adjust this parameter during pipeline initialization:

from speech_to_speech.connections.local_audio_streamer import LocalAudioStreamer

# Reduce latency at cost of CPU cycles

streamer = LocalAudioStreamer(
    samplerate=16000,
    blocksize=256,  # 16 ms at 16 kHz

    input_queue=input_queue,
    output_queue=output_queue
)

Detecting Output Queue Underruns

When output_queue is empty, the audio callback outputs silence (outdata[:] = 0), perceived as dropped frames. Monitor queue depth to catch this condition:

import threading
import logging

def monitor_queues(pipeline, interval=0.2):
    """Log queue sizes to detect back-pressure."""
    logger = logging.getLogger(__name__)
    
    def _log():
        in_size = pipeline.input_queue.qsize()
        out_size = pipeline.output_queue.qsize()
        logger.debug("Queue sizes – input: %d, output: %d", in_size, out_size)
        
        if out_size == 0:
            logger.warning("Output queue empty – frames may be dropped")
        
        threading.Timer(interval, _log).start()
    
    _log()

Debugging WebSocket Transport and Network Jitter

The WebsocketStreamer in src/speech_to_speech/connections/websocket_streamer.py serializes audio chunks to the OpenAI Realtime WebSocket. The AUDIO_PTIME = 0.02 constant in webrtc_session.py defines the expected 20 ms frame duration.

When network jitter exceeds this window, frames arrive incomplete or out of order. The streamer logs warnings when "WebSocket frame boundaries are not aligned" (around line 129).

Enable debug logging for the audio handler submodule:

export LOG_LEVEL=DEBUG

# Specifically for frame-level tracing:

python -c "
import logging
logging.getLogger('speech_to_speech.api.openai_realtime.handlers.audio').setLevel(logging.DEBUG)
"

Measuring Resampling Latency

The WebrtcAudioTrack class buffers and resamples inbound 48 kHz RTP frames to 16 kHz for VAD/STT processing. Verify resampler configuration:

import av

# Check for unexpected buffering in the resampler

resampler = av.AudioResampler(
    format='s16',
    layout='mono',
    rate=16000,
    frame_size=960  # 20 ms at 48 kHz input

)

Buffer underruns while awaiting RTP packets create audible gaps. The resampler's frame_size should match the expected input frame rate to minimize internal queuing.

Identifying VAD and STT Bottlenecks

VADIterator Queue Behavior

VADIterator in src/speech_to_speech/VAD/vad_iterator.py runs in a dedicated thread consuming from input_queue. If VAD processing lags behind capture, the queue grows unbounded and upstream latency increases.

Check for VAD-induced back-pressure by timestamping speech segment boundaries. The iterator emits events at speech start and end—correlate these timestamps with the original capture time.

STT Inference Stalls

Speech-to-text handlers like MLXAudioWhisperHandler perform the heaviest compute in the pipeline. Long-running inference blocks the entire chain, eventually emptying output_queue and causing playback silence.

Run the STT benchmark in isolation:

python scripts/benchmark_stt.py --model mlx-whisper-large-v3

Analyzing TTS Streaming Latency

The Qwen3TTSHandler in src/speech_to_speech/TTS/qwen3_tts_handler.py implements _log_first_audio_latency, which measures the critical path from when speech stops to when the first audio chunk emits.

from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler
import time

handler = Qwen3TTSHandler(model_path="Qwen/Qwen2.5-TTS-0.5B")

# Simulate input with known stop time

class DummyTTSInput:
    speech_stopped_at_s = time.perf_counter() - 0.05

handler._log_first_audio_latency(DummyTTSInput())

# Expected log: "First audio latency: 0.052 s (speech stop→first audio frame)"

High first-frame latency indicates:

• Missing model warm-up (CUDA/MLX graph capture on first use)
• Incorrect list_play_chunk_size parameter
• GPU contention or thermal throttling

Pre-warm models during pipeline startup to eliminate cold-start latency.

Systematic Debugging Workflow

1. Enable global debug logging

export LOG_LEVEL=DEBUG
python -m speech_to_speech.demo

2. Inject queue monitoring (temporary patch to s2s_pipeline.py):

# After queue creation in SpeechToSpeechPipeline.__init__

from threading import Timer

def log_queue_sizes(pipeline):
    logger = logging.getLogger(__name__)
    def _inner():
        logger.debug(
            "Queues – in: %s, out: %s",
            pipeline.input_queue.qsize(),
            pipeline.output_queue.qsize(),
        )
        Timer(0.5, _inner).start()
    _inner()

# Call after pipeline instantiation

log_queue_sizes(pipeline)

3. Run component benchmarks to isolate model-specific issues:

# TTS latency only

python scripts/benchmark_tts.py --model qwen3-tts-0.5b --iterations 100

# STT latency only  

python scripts/benchmark_stt.py --model mlx-whisper-large-v3 --duration 30

4. Correlate timestamps across log lines:
   • LocalAudioStreamer callback timestamps
   • VADIterator speech start/end events
   • STT request/response durations
   • Qwen3TTSHandler._log_first_audio_latency values

Key Configuration Parameters

Parameter	Location	Default	Tuning Guidance
blocksize	LocalAudioStreamer	512	Reduce to 256 for lower latency; increase if CPU-bound
AUDIO_PTIME	webrtc_session.py	0.02 s	Fixed by protocol; ensure network RTT < 40 ms
list_play_chunk_size	TTS handlers	Model-dependent	Reduce to minimize buffering; increase if audio glitches
frame_size	av.AudioResampler	960	Match input sample rate × frame duration

Summary

• Capture latency: Tune blocksize in LocalAudioStreamer (default 512 → ~32 ms at 16 kHz)
• Queue back-pressure: Monitor input_queue.qsize() and output_queue.qsize() to find slow consumers
• Network issues: Watch for frame boundary misalignment warnings in WebsocketStreamer
• Model inference: Check _log_first_audio_latency in TTS handlers and run benchmark_stt.py/benchmark_tts.py for isolation testing
• Resampling delay: Verify WebrtcAudioTrack resampler configuration matches expected input rates

Frequently Asked Questions

How do I enable debug logging for the speech-to-speech pipeline?

Set the LOG_LEVEL environment variable to DEBUG before running. All module-level loggers under speech_to_speech.* respect this setting, emitting timestamps for capture, VAD, STT, and TTS events. For frame-level WebSocket tracing, additionally configure speech_to_speech.api.openai_realtime.handlers.audio to DEBUG level in Python.

Why am I hearing silent gaps during playback?

Silent gaps typically indicate an empty output_queue in LocalAudioStreamer. This occurs when downstream stages (VAD, STT, LLM, or TTS) cannot produce audio chunks fast enough. Monitor output_queue.qsize() to confirm; persistent emptiness points to a bottleneck in model inference or queue processing.

What causes high first-frame latency in TTS?

The _log_first_audio_latency measurement in Qwen3TTSHandler captures model warm-up time, including CUDA/MLX graph compilation and memory allocation. Pre-warm the model during pipeline initialization by running a dummy inference, or check for GPU graph capture failures in the logs.

How do I distinguish network jitter from processing delays?

Network jitter in WebsocketStreamer produces specific log warnings about frame boundary misalignment and AUDIO_PTIME violations. Processing delays appear as growing queue sizes in input_queue without corresponding network-level errors. Enable debug logging for both websocket_streamer and webrtc_session modules to isolate the root cause.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}