# How to Debug Audio Pipeline Latency Issues End-to-End in Hugging Face Speech-to-Speech

> Debug audio pipeline latency end-to-end in Hugging Face speech-to-speech. Tune VAD chunk sizes, enable live transcription, and select streaming TTS modes for lower latency.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-03

---

**The Hugging Face `speech-to-speech` pipeline reduces end-to-end latency by tuning VAD chunk sizes, enabling live transcription, selecting streaming TTS modes, and using local LLM backends like `mlx-lm` or FP16 Transformers.**

Debugging audio pipeline latency in the `huggingface/speech-to-speech` repository requires understanding its four-stage architecture and the specific knobs available in each component. This guide walks through the exact source code locations, CLI flags, and measurement techniques needed to identify and eliminate bottlenecks in real-time speech-to-speech systems.

## Understanding the Four-Stage Pipeline Architecture

The Speech-to-Speech pipeline is a **four-stage cascade** that runs each component in its own thread and connects them with lock-free queues:

| Stage | Responsibility | Default Implementation | Core File |
|------|----------------|------------------------|-----------|
| **VAD** | Detects speech boundaries and turn-taking | Silero VAD v5 | [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) |
| **STT** | Transcribes spoken turns with optional live partial results | Parakeet TDT (default) — also Whisper, Faster-Whisper | [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) |
| **LLM** | Generates textual responses via streaming | OpenAI-compatible Responses-API, Transformers, or `mlx-lm` | [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) |
| **TTS** | Synthesizes audio from LLM output and streams it back | Qwen3-TTS (default) — also Kokoro, Pocket, ChatTTS, Facebook MMS | [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) |

All stages are assembled in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) through `_build_pipeline_handlers` → `build_pipeline`. The pipeline initializes queues via `initialize_queues_and_events` (lines 64-71) that carry data between stages.

## Identifying Where Latency Is Introduced

### Stage 1: Audio Capture → VAD

The microphone thread (`LocalAudioStreamer` or `WebSocketStreamer`) writes raw PCM to `recv_audio_chunks_queue`. Latency here depends on:

- **Chunk size**: Controlled by `--chunk_size` in [`socket_receiver_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/socket_receiver_arguments.py)
- **VAD processing interval**: Set via `vad_handler_kwargs.realtime_processing_pause`

### Stage 2: VAD → STT

VAD emits `VADOutItem` objects (see `VADHandler` at line 18 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)). When **live transcription** is enabled (`module_kwargs.enable_live_transcription`), the STT handler streams partial results. STT latency is dominated by model inference time and internal chunking strategies — for example, Whisper buffers 0.5 seconds of audio before each forward pass.

### Stage 3: STT → LLM

The `TranscriptionNotifier` forwards `STTOutItem` to the LLM queue. LLM latency is the most variable component. The code tracks it in [`api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/service.py) (line 147 comment: "latency tts, llm, vad, stt").

### Stage 4: LLM → TTS

`LMOutputProcessor` inserts a `text_output_queue` for optional text-only events, then passes `TTSInItem` to the TTS handler. TTS latency is measured in `Qwen3TTSHandler._log_first_audio_latency` (lines 828-835), which logs the time between `speech_stopped_at_s` (LLM finished streaming) and first audio chunk emission.

### Stage 5: TTS → Playback

The final audio queue (`send_audio_chunks_queue`) is drained by the communication layer. Playback clients may add their own buffering — for example, the browser WebSocket demo buffers several frames for smooth output.

## Essential Debugging Steps for Audio Pipeline Latency

### Enable Detailed Timing Logs

Set `--log_level debug` to activate granular per-stage logging. The `PipelineLogFilter` adds a `pipeline_prefix` to each log line, making it easy to spot which stage produced a delay.

```bash
speech-to-speech --log_level debug

```

Reference: `setup_logger` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), lines 34-44.

### Measure Per-Stage Latency Programmatically

The realtime service aggregates latency statistics (mean, max, p90) for all four stages. Access these via `RealtimeService.latency_report`:

```python
from speech_to_speech.api.openai_realtime.service import RealtimeService

service = RealtimeService(...)

# After conversation:

print(service.latency_report())  # {'vad': {...}, 'stt': {...}, 'llm': {...}, 'tts': {...}}

```

Reference: [`api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/service.py), line 147 comment.

### Tune VAD Chunk Size

Reduce `--chunk_size` (socket mode) or `--realtime_processing_pause` (VAD argument) to lower time spent waiting for audio buffers:

```bash
speech-to-speech --chunk_size 256 --realtime_processing_pause 0.01

```

Reference: `VADHandlerArguments` in [`arguments_classes/vad_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/arguments_classes/vad_arguments.py).

### Adjust STT Buffering

For Whisper-based handlers, `max_chunk_ms` controls audio batching before forward pass. Reduce it to lower transcription latency at higher compute cost:

```bash
speech-to-speech --stt whisper --whisper_stt_max_chunk_ms 250  # default: 500ms

```

Reference: `WhisperSTTHandler` implementation in [`STT/whisper_stt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/STT/whisper_stt_handler.py).

### Enable Live Transcription

Stream partial transcripts to the LLM, allowing generation to start while the user is still speaking:

```bash
speech-to-speech --enable_live_transcription

```

**Important**: On Apple Silicon with multiple pipelines, the code disables live transcription automatically (see lines 70-76 of [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py)) to prevent MLX lock contention.

### Select a Low-Latency LLM Backend

| Backend | Best For | Activation |
|---------|----------|------------|
| **mlx-lm** | Apple Silicon, single-forward-pass pipeline | `--llm_backend mlx-lm` |
| **transformers** | CUDA machines with FP16 | `--llm_backend transformers` + FP16 |
| **Responses-API** | Remote inference with streaming | Choose streaming-capable models: `gpt-4o-mini`, `gpt-oss-20b` |

Reference: `get_llm_handler` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py), lines 91-105.

### Use Streaming TTS Mode

Qwen3-TTS supports **non-streaming mode** (`--qwen3_tts_non_streaming_mode True`). For lowest latency, keep this **false** so frames emit immediately:

```bash
speech-to-speech --tts qwen3 --qwen3_tts_non_streaming_mode False

```

First-audio latency is logged in `Qwen3TTSHandler._log_first_audio_latency` (lines 828-835).

### Profile Hardware Contention

On macOS with Apple Silicon, the global MLX lock serializes inference ([`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)). Running multiple pipelines (`--num_pipelines > 1`) triggers automatic live transcription disable (lines 70-76). For lowest latency, use a single pipeline or avoid lock-contended models.

## Complete Debugging Workflow Example

```bash

# Maximum diagnostics, optimized for low latency

speech-to-speech \
  --log_level debug \
  --enable_live_transcription \
  --chunk_size 256 \
  --qwen3_tts_non_streaming_mode False \
  --stt whisper \
  --whisper_stt_max_chunk_ms 200 \
  --llm_backend mlx-lm \
  --device cuda \
  --num_pipelines 1

```

## Latency Troubleshooting Checklist

Use this structured approach when debugging audio pipeline latency issues end-to-end:

1. **Log Level** — Set `--log_level debug` for granular visibility
2. **Chunk Size** — Reduce `--chunk_size` or `--realtime_processing_pause`
3. **Live Transcription** — Enable unless on macOS with >1 pipeline
4. **STT Buffer** — Lower `max_chunk_ms` for Whisper handlers
5. **LLM Backend** — Prefer `mlx-lm` or FP16 Transformers for sub-second response
6. **TTS Streaming** — Keep `--qwen3_tts_non_streaming_mode False`
7. **Hardware Utilization** — Verify GPU/CPU saturation with `nvidia-smi` or Activity Monitor

## Key Source Files for Latency Debugging

| File | Relevance |
|------|-----------|
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Orchestrates pipeline; defines queues, event flow, latency reporter attachment |
| [`src/speech_to_speech/connections/local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/local_audio_streamer.py) | Microphone capture; controls initial audio chunk sizing |
| [`src/speech_to_speech/VAD/vad_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/VAD/vad_handler.py) | VAD implementation and realtime pause tuning |
| [`src/speech_to_speech/STT/parakeet_tdt_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/parakeet_tdt_handler.py) | Default STT; contains live-transcription loop |
| [`src/speech_to_speech/LLM/responses_api_language_model.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/LLM/responses_api_language_model.py) | Streaming LLM responses; network round-trip dominates |
| [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | Audio streaming; first-audio latency logging (lines 828-835) |
| [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) | Per-stage latency aggregation (line 147) |
| `src/speech_to_speech/arguments_classes/*` | All CLI flags for buffering, chunk size, live transcription |
| [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) | MLX serialization on macOS |
| [`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py) | Experimental TTS latency measurement |

## Summary

- The `speech-to-speech` pipeline uses **four threaded stages** (VAD → STT → LLM → TTS) connected by lock-free queues
- **Enable debug logging** and use `RealtimeService.latency_report()` to pinpoint slow stages
- **Reduce buffering** at VAD (`--chunk_size`, `--realtime_processing_pause`) and STT (`max_chunk_ms`) levels
- **Enable live transcription** to overlap STT and LLM work, except on Apple Silicon with multiple pipelines
- **Prefer streaming modes**: keep `--qwen3_tts_non_streaming_mode False` and select streaming-compatible LLM backends
- **Monitor hardware contention**: MLX lock on macOS can artificially inflate latency when running multiple pipelines

## Frequently Asked Questions

### How do I measure end-to-end latency in the speech-to-speech pipeline?

Use the built-in latency reporter. Instantiate `RealtimeService` from [`api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/api/openai_realtime/service.py) and call `latency_report()` after running conversation. For CLI debugging, add `--log_level debug` to see per-stage timing in the logs.

### Why is my STT latency higher than expected with Whisper?

Whisper handlers default to buffering 500ms of audio before inference. Override with `--whisper_stt_max_chunk_ms 200` or lower. This increases GPU/CPU utilization but reduces transcription delay. Also verify that live transcription is enabled (`--enable_live_transcription`) to stream partial results.

### Can I run multiple pipelines for lower latency?

Generally no — multiple pipelines increase contention. On Apple Silicon, the MLX lock ([`utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/utils/mlx_lock.py)) serializes inference, and the code automatically disables live transcription when `--num_pipelines > 1`. For lowest latency, use a single pipeline with optimized buffering instead.