# How to Monitor and Debug Latency Issues in the Speech-to-Speech Pipeline

> Effectively monitor and debug speech-to-speech pipeline latency. Learn to track TTFA and RTF using built-in measurements for faster, more responsive applications.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: tutorial
- Published: 2026-08-10

---

**The huggingface/speech-to-speech pipeline tracks latency through built-in `time.perf_counter()` measurements that emit TTFA (Time-to-First-Audio) and RTF (Real-Time Factor) logs in `Qwen3TTSHandler._stream` and per-chunk inference timings in [`smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_progressive_streaming.py).**

Latency optimization requires visibility into each stage of the speech-to-speech pipeline. The huggingface/speech-to-speech repository implements fine-grained timing instrumentation across its VAD, STT, LLM, and TTS components, allowing developers to identify bottlenecks without external profiling tools.

## Key Latency Metrics Defined

The pipeline captures four critical measurements that reveal where time is spent:

| Metric | Definition | Source Location |
|--------|-----------|-----------------|
| **TTFA** (Time-to-First-Audio) | Seconds from TTS request to first audio chunk output | [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) line 706 |
| **RTF** (Real-Time Factor) | Audio duration divided by generation time; values < 1 indicate slower-than-real-time synthesis | Same `_stream` method, line 39 |
| **First-Audio Latency (Pocket TTS)** | Time-to-first-audio for the lightweight Pocket TTS variant | [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) line 164 |
| **STT Inference Time** | Milliseconds per audio chunk during streaming transcription | [`src/speech_to_speech/STT/smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/smart_progressive_streaming.py) line 305 |

## How Timing Measurements Work

### Stage-Level Instrumentation

Each pipeline handler follows a consistent three-marker pattern using Python's `time.perf_counter()`:

1. **Start marker** — recorded immediately upon request entry
2. **First-output marker** — triggers TTFA calculation and logging
3. **Completion marker** — enables RTF computation from total samples and elapsed wall time

In `Qwen3TTSHandler._stream`, the implementation looks like this:

```python
start = perf_counter()

# ... model inference ...

logger.info(f"Qwen3-TTS TTFA: {perf_counter() - start:.2f}s ({label})")

# ... stream all chunks ...

generation_time = perf_counter() - start
audio_duration = total_samples / PIPELINE_SR
rtf = audio_duration / generation_time
logger.info(f"Qwen3-TTS generated {audio_duration:.2f}s audio in {generation_time:.2f}s (RTF: {rtf:.2f}, {label})")

```

### Connection-Wide Aggregation

The `GlobalUsageMetrics` dataclass in [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) (line 51) provides session-level counters. While currently focused on token and audio volume tracking, this structure can be extended to accumulate latency histograms across multiple requests.

## Step-by-Step Debugging Workflow

### 1. Enable Verbose Logging

Set the logging level before server initialization to capture all timing events:

```python
import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s | %(levelname)s | %(message)s'
)

```

### 2. Parse TTFA and RTF from Live Logs

Extract structured latency data using regex filters:

```python
import re
import logging

TTFA_PATTERN = re.compile(r'Qwen3-TTS TTFA: ([\d.]+)s')
RTF_PATTERN = re.compile(r'RTF: ([\d.]+)')

class LatencyFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        if match := TTFA_PATTERN.search(msg):
            record.ttfa_seconds = float(match.group(1))
            print(f"⚡ TTFA: {record.ttfa_seconds:.3f}s")
        if match := RTF_PATTERN.search(msg):
            record.rtf_value = float(match.group(1))
            print(f"📊 RTF: {record.rtf_value:.2f}")
        return True  # Allow record through to other handlers

logging.getLogger().handlers[0].addFilter(LatencyFilter())

```

### 3. Correlate Cross-Stage Timestamps

Every pipeline event carries a `created_at_s` timestamp. Compute inter-stage gaps:

- `SpeechStartedEvent.created_at_s` → `STTCompleteEvent.created_at_s` = VAD + STT latency
- `STTCompleteEvent.created_at_s` → `LLMResponseEvent.created_at_s` = LLM generation time
- `LLMResponseEvent.created_at_s` → `AudioOutputStartEvent.created_at_s` = TTS TTFA

### 4. Run Benchmark Baselines

Use the provided benchmark scripts for controlled measurements:

```bash

# TTS benchmark: reports warm-up, TTFA, and total processing time

python -m scripts.benchmark_tts \
    --model qwen3_tts \
    --text "This is a controlled latency measurement."

# STT benchmark: measures per-chunk inference overhead

python -m scripts.benchmark_stt \
    --audio_path sample_16khz.wav

```

## Tuning Parameters Based on Metrics

### High TTFA (> 500ms)

- **Reduce `streaming_chunk_size`** — controls initial buffer before first output; defined in [`src/speech_to_speech/arguments_classes/chat_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/chat_tts_arguments.py) line 17
- **Implement model warm-up** — execute a dummy request before serving live traffic to preload weights and kernels

### Low RTF (< 1.0, slower than real-time)

- **Increase `blocksize`** — larger audio blocks reduce per-chunk overhead
- **Disable unnecessary resampling** — check post-processing pipeline in your TTS handler
- **Profile GPU utilization** — RTF < 1 often indicates compute-bound generation

### STT Streaming Delays

Check [`smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_progressive_streaming.py) for per-chunk inference logs. If `Inference X.XX ms` exceeds your target latency budget, consider:
- Reducing model precision (FP16 → INT8)
- Adjusting `vad_sensitivity` to trigger transcription less frequently

## Key Source Files for Latency Analysis

| Component | File Path | Primary Functions |
|-----------|-----------|-------------------|
| Qwen3-TTS timing | [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | `_stream()` — TTFA and RTF logging |
| Pocket TTS timing | [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) | `_stream()` — first-audio measurement |
| STT streaming | [`src/speech_to_speech/STT/smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/STT/smart_progressive_streaming.py) | `smart_progressive_streaming()`, `smart_turn()` |
| Session metrics | [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) | `GlobalUsageMetrics`, `RealtimeService.unregister()` |
| TTS arguments | [`src/speech_to_speech/arguments_classes/chat_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/chat_tts_arguments.py) | `streaming_chunk_size` parameter |

## Summary

- **TTFA and RTF logs** in [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) provide immediate visibility into TTS latency characteristics
- **Per-chunk STT timing** in [`smart_progressive_streaming.py`](https://github.com/huggingface/speech-to-speech/blob/main/smart_progressive_streaming.py) reveals transcription bottlenecks during streaming
- **`GlobalUsageMetrics`** offers extensible infrastructure for session-wide latency aggregation
- **Benchmark scripts** ([`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py), [`scripts/benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py)) establish reproducible baselines
- **Configuration tuning** of `streaming_chunk_size` and `blocksize` directly impacts observed latency metrics

## Frequently Asked Questions

### How do I enable DEBUG-level logging to see all latency measurements?

Set `LOG_LEVEL=DEBUG` as an environment variable or configure `logging.basicConfig(level=logging.DEBUG)` before importing the speech-to-speech modules. This exposes per-chunk STT inference times and verbose TTS generation details.

### What TTFA value indicates a properly optimized TTS setup?

TTFA below **200-300ms** is achievable with warm models and appropriate `streaming_chunk_size` values. Values consistently above 500ms suggest model loading overhead or excessively large initial buffers.

### Can I export latency metrics to Prometheus or another metrics system?

Yes. Extend the `GlobalUsageMetrics` class in [`src/speech_to_speech/api/openai_realtime/service.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/api/openai_realtime/service.py) to compute histograms from the per-handler logs, then expose them through your preferred metrics endpoint during `RealtimeService.unregister()` cleanup.

### Why is my RTF greater than 1.0 but TTFA is acceptable?

RTF > 1 indicates that total generation time exceeds audio duration, causing audible gaps or stuttering despite quick first-chunk response. This typically stems from slow token generation or post-processing bottlenecks after the initial output. Increase `blocksize` or optimize the synthesis backend.