# How to Debug Audio Pipeline Latency and Dropped Frames in Speech-to-Speech: A Complete Guide

> Debug audio pipeline latency and dropped frames in speech-to-speech. Monitor queues, log debug info, and inspect TTS handlers to pinpoint root causes like inference, network issues, or blocksize problems.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-06

---

**Set `LOG_LEVEL=DEBUG`, monitor queue sizes in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), and inspect `_log_first_audio_latency` in TTS handlers to identify whether latency stems from capture blocksize, model inference, network jitter, or queue back-pressure.**

The Hugging Face `speech-to-speech` repository implements a real-time audio pipeline connecting microphone capture, voice-activity detection, speech-to-text inference, large language model generation, and text-to-speech playback. Debugging audio pipeline latency and dropped frames requires tracing timing across six interconnected stages, each with distinct failure modes and diagnostic hooks.

## Understanding the Pipeline Architecture

The system processes audio through a chain of queues and background threads orchestrated by [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py). Latency and frame loss manifest differently depending on which stage becomes the bottleneck.

| Stage | Core Module | Latency Risk |
|-------|-------------|--------------|
| **Capture / Playback** | `LocalAudioStreamer` ([`local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/local_audio_streamer.py)) | `blocksize` parameter directly sets capture-to-playback delay |
| **WebSocket Transport** | `WebsocketStreamer` ([`websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/websocket_streamer.py)) | Network jitter causes frame misalignment beyond `AUDIO_PTIME = 0.02` |
| **Resampling** | `WebrtcAudioTrack` ([`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py)) | 48 kHz → 16 kHz conversion adds buffering delay |
| **Voice-Activity Detection** | `VADIterator` ([`vad_iterator.py`](https://github.com/huggingface/speech-to-speech/blob/main/vad_iterator.py)) | Queue back-pressure when processing lags behind capture |
| **STT Inference** | Handlers like `MLXAudioWhisperHandler` | Model inference stalls downstream components |
| **TTS Inference** | Handlers like `Qwen3TTSHandler` | First-frame latency from model warm-up or GPU graph capture |

## Diagnosing Capture and Playback Issues

### Blocksize Configuration in LocalAudioStreamer

The `LocalAudioStreamer` class in [`src/speech_to_speech/connections/local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/local_audio_streamer.py) uses `sounddevice.Stream` with a default `blocksize=512` samples. At 16 kHz sampling rate, this equals approximately 32 ms of audio per callback.

**Too large**: Increases end-to-end latency.  
**Too small**: Raises CPU utilization and risk of dropouts.

Adjust this parameter during pipeline initialization:

```python
from speech_to_speech.connections.local_audio_streamer import LocalAudioStreamer

# Reduce latency at cost of CPU cycles

streamer = LocalAudioStreamer(
    samplerate=16000,
    blocksize=256,  # 16 ms at 16 kHz

    input_queue=input_queue,
    output_queue=output_queue
)

```

### Detecting Output Queue Underruns

When `output_queue` is empty, the audio callback outputs silence (`outdata[:] = 0`), perceived as dropped frames. Monitor queue depth to catch this condition:

```python
import threading
import logging

def monitor_queues(pipeline, interval=0.2):
    """Log queue sizes to detect back-pressure."""
    logger = logging.getLogger(__name__)
    
    def _log():
        in_size = pipeline.input_queue.qsize()
        out_size = pipeline.output_queue.qsize()
        logger.debug("Queue sizes – input: %d, output: %d", in_size, out_size)
        
        if out_size == 0:
            logger.warning("Output queue empty – frames may be dropped")
        
        threading.Timer(interval, _log).start()
    
    _log()

```

## Debugging WebSocket Transport and Network Jitter

The `WebsocketStreamer` in [`src/speech_to_speech/connections/websocket_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/websocket_streamer.py) serializes audio chunks to the OpenAI Realtime WebSocket. The `AUDIO_PTIME = 0.02` constant in [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) defines the expected 20 ms frame duration.

When network jitter exceeds this window, frames arrive incomplete or out of order. The streamer logs warnings when `"WebSocket frame boundaries are not aligned"` (around line 129).

Enable debug logging for the audio handler submodule:

```bash
export LOG_LEVEL=DEBUG

# Specifically for frame-level tracing:

python -c "
import logging
logging.getLogger('speech_to_speech.api.openai_realtime.handlers.audio').setLevel(logging.DEBUG)
"

```

## Measuring Resampling Latency

The `WebrtcAudioTrack` class buffers and resamples inbound 48 kHz RTP frames to 16 kHz for VAD/STT processing. Verify resampler configuration:

```python
import av

# Check for unexpected buffering in the resampler

resampler = av.AudioResampler(
    format='s16',
    layout='mono',
    rate=16000,
    frame_size=960  # 20 ms at 48 kHz input

)

```

Buffer underruns while awaiting RTP packets create audible gaps. The resampler's `frame_size` should match the expected input frame rate to minimize internal queuing.

## Identifying VAD and STT Bottlenecks

### VADIterator Queue Behavior

`VADIterator` in [`src/speech_to_speech/VAD/vad_iterator.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_iterator.py) runs in a dedicated thread consuming from `input_queue`. If VAD processing lags behind capture, the queue grows unbounded and upstream latency increases.

Check for VAD-induced back-pressure by timestamping speech segment boundaries. The iterator emits events at speech start and end—correlate these timestamps with the original capture time.

### STT Inference Stalls

Speech-to-text handlers like `MLXAudioWhisperHandler` perform the heaviest compute in the pipeline. Long-running inference blocks the entire chain, eventually emptying `output_queue` and causing playback silence.

Run the STT benchmark in isolation:

```bash
python scripts/benchmark_stt.py --model mlx-whisper-large-v3

```

## Analyzing TTS Streaming Latency

The `Qwen3TTSHandler` in [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) implements `_log_first_audio_latency`, which measures the critical path from when speech stops to when the first audio chunk emits.

```python
from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler
import time

handler = Qwen3TTSHandler(model_path="Qwen/Qwen2.5-TTS-0.5B")

# Simulate input with known stop time

class DummyTTSInput:
    speech_stopped_at_s = time.perf_counter() - 0.05

handler._log_first_audio_latency(DummyTTSInput())

# Expected log: "First audio latency: 0.052 s (speech stop→first audio frame)"

```

**High first-frame latency indicates:**
- Missing model warm-up (CUDA/MLX graph capture on first use)
- Incorrect `list_play_chunk_size` parameter
- GPU contention or thermal throttling

Pre-warm models during pipeline startup to eliminate cold-start latency.

## Systematic Debugging Workflow

1. **Enable global debug logging**

```bash
export LOG_LEVEL=DEBUG
python -m speech_to_speech.demo

```

2. **Inject queue monitoring** (temporary patch to [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)):

```python

# After queue creation in SpeechToSpeechPipeline.__init__

from threading import Timer

def log_queue_sizes(pipeline):
    logger = logging.getLogger(__name__)
    def _inner():
        logger.debug(
            "Queues – in: %s, out: %s",
            pipeline.input_queue.qsize(),
            pipeline.output_queue.qsize(),
        )
        Timer(0.5, _inner).start()
    _inner()

# Call after pipeline instantiation

log_queue_sizes(pipeline)

```

3. **Run component benchmarks** to isolate model-specific issues:

```bash

# TTS latency only

python scripts/benchmark_tts.py --model qwen3-tts-0.5b --iterations 100

# STT latency only  

python scripts/benchmark_stt.py --model mlx-whisper-large-v3 --duration 30

```

4. **Correlate timestamps** across log lines:
   - `LocalAudioStreamer` callback timestamps
   - `VADIterator` speech start/end events
   - STT request/response durations
   - `Qwen3TTSHandler._log_first_audio_latency` values

## Key Configuration Parameters

| Parameter | Location | Default | Tuning Guidance |
|-----------|----------|---------|---------------|
| `blocksize` | `LocalAudioStreamer` | 512 | Reduce to 256 for lower latency; increase if CPU-bound |
| `AUDIO_PTIME` | [`webrtc_session.py`](https://github.com/huggingface/speech-to-speech/blob/main/webrtc_session.py) | 0.02 s | Fixed by protocol; ensure network RTT < 40 ms |
| `list_play_chunk_size` | TTS handlers | Model-dependent | Reduce to minimize buffering; increase if audio glitches |
| `frame_size` | `av.AudioResampler` | 960 | Match input sample rate × frame duration |

## Summary

- **Capture latency**: Tune `blocksize` in `LocalAudioStreamer` (default 512 → ~32 ms at 16 kHz)
- **Queue back-pressure**: Monitor `input_queue.qsize()` and `output_queue.qsize()` to find slow consumers
- **Network issues**: Watch for frame boundary misalignment warnings in `WebsocketStreamer`
- **Model inference**: Check `_log_first_audio_latency` in TTS handlers and run [`benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_stt.py)/[`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py) for isolation testing
- **Resampling delay**: Verify `WebrtcAudioTrack` resampler configuration matches expected input rates

## Frequently Asked Questions

### How do I enable debug logging for the speech-to-speech pipeline?

Set the `LOG_LEVEL` environment variable to `DEBUG` before running. All module-level loggers under `speech_to_speech.*` respect this setting, emitting timestamps for capture, VAD, STT, and TTS events. For frame-level WebSocket tracing, additionally configure `speech_to_speech.api.openai_realtime.handlers.audio` to DEBUG level in Python.

### Why am I hearing silent gaps during playback?

Silent gaps typically indicate an empty `output_queue` in `LocalAudioStreamer`. This occurs when downstream stages (VAD, STT, LLM, or TTS) cannot produce audio chunks fast enough. Monitor `output_queue.qsize()` to confirm; persistent emptiness points to a bottleneck in model inference or queue processing.

### What causes high first-frame latency in TTS?

The `_log_first_audio_latency` measurement in `Qwen3TTSHandler` captures model warm-up time, including CUDA/MLX graph compilation and memory allocation. Pre-warm the model during pipeline initialization by running a dummy inference, or check for GPU graph capture failures in the logs.

### How do I distinguish network jitter from processing delays?

Network jitter in `WebsocketStreamer` produces specific log warnings about frame boundary misalignment and `AUDIO_PTIME` violations. Processing delays appear as growing queue sizes in `input_queue` without corresponding network-level errors. Enable debug logging for both `websocket_streamer` and `webrtc_session` modules to isolate the root cause.