How to Monitor and Debug Latency Issues in the Speech-to-Speech Pipeline

The huggingface/speech-to-speech pipeline tracks latency through built-in time.perf_counter() measurements that emit TTFA (Time-to-First-Audio) and RTF (Real-Time Factor) logs in Qwen3TTSHandler._stream and per-chunk inference timings in smart_progressive_streaming.py.

Latency optimization requires visibility into each stage of the speech-to-speech pipeline. The huggingface/speech-to-speech repository implements fine-grained timing instrumentation across its VAD, STT, LLM, and TTS components, allowing developers to identify bottlenecks without external profiling tools.

Key Latency Metrics Defined

The pipeline captures four critical measurements that reveal where time is spent:

Metric	Definition	Source Location
TTFA (Time-to-First-Audio)	Seconds from TTS request to first audio chunk output	src/speech_to_speech/TTS/qwen3_tts_handler.py line 706
RTF (Real-Time Factor)	Audio duration divided by generation time; values < 1 indicate slower-than-real-time synthesis	Same _stream method, line 39
First-Audio Latency (Pocket TTS)	Time-to-first-audio for the lightweight Pocket TTS variant	src/speech_to_speech/TTS/pocket_tts_handler.py line 164
STT Inference Time	Milliseconds per audio chunk during streaming transcription	src/speech_to_speech/STT/smart_progressive_streaming.py line 305

How Timing Measurements Work

Stage-Level Instrumentation

Each pipeline handler follows a consistent three-marker pattern using Python's time.perf_counter():

1. Start marker — recorded immediately upon request entry
2. First-output marker — triggers TTFA calculation and logging
3. Completion marker — enables RTF computation from total samples and elapsed wall time

In Qwen3TTSHandler._stream, the implementation looks like this:

start = perf_counter()

# ... model inference ...

logger.info(f"Qwen3-TTS TTFA: {perf_counter() - start:.2f}s ({label})")

# ... stream all chunks ...

generation_time = perf_counter() - start
audio_duration = total_samples / PIPELINE_SR
rtf = audio_duration / generation_time
logger.info(f"Qwen3-TTS generated {audio_duration:.2f}s audio in {generation_time:.2f}s (RTF: {rtf:.2f}, {label})")

Connection-Wide Aggregation

The GlobalUsageMetrics dataclass in src/speech_to_speech/api/openai_realtime/service.py (line 51) provides session-level counters. While currently focused on token and audio volume tracking, this structure can be extended to accumulate latency histograms across multiple requests.

Step-by-Step Debugging Workflow

1. Enable Verbose Logging

Set the logging level before server initialization to capture all timing events:

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s | %(levelname)s | %(message)s'
)

2. Parse TTFA and RTF from Live Logs

Extract structured latency data using regex filters:

import re
import logging

TTFA_PATTERN = re.compile(r'Qwen3-TTS TTFA: ([\d.]+)s')
RTF_PATTERN = re.compile(r'RTF: ([\d.]+)')

class LatencyFilter(logging.Filter):
    def filter(self, record):
        msg = record.getMessage()
        if match := TTFA_PATTERN.search(msg):
            record.ttfa_seconds = float(match.group(1))
            print(f"⚡ TTFA: {record.ttfa_seconds:.3f}s")
        if match := RTF_PATTERN.search(msg):
            record.rtf_value = float(match.group(1))
            print(f"📊 RTF: {record.rtf_value:.2f}")
        return True  # Allow record through to other handlers

logging.getLogger().handlers[0].addFilter(LatencyFilter())

3. Correlate Cross-Stage Timestamps

Every pipeline event carries a created_at_s timestamp. Compute inter-stage gaps:

• SpeechStartedEvent.created_at_s → STTCompleteEvent.created_at_s = VAD + STT latency
• STTCompleteEvent.created_at_s → LLMResponseEvent.created_at_s = LLM generation time
• LLMResponseEvent.created_at_s → AudioOutputStartEvent.created_at_s = TTS TTFA

4. Run Benchmark Baselines

Use the provided benchmark scripts for controlled measurements:

# TTS benchmark: reports warm-up, TTFA, and total processing time

python -m scripts.benchmark_tts \
    --model qwen3_tts \
    --text "This is a controlled latency measurement."

# STT benchmark: measures per-chunk inference overhead

python -m scripts.benchmark_stt \
    --audio_path sample_16khz.wav

Tuning Parameters Based on Metrics

High TTFA (> 500ms)

• Reduce streaming_chunk_size — controls initial buffer before first output; defined in src/speech_to_speech/arguments_classes/chat_tts_arguments.py line 17
• Implement model warm-up — execute a dummy request before serving live traffic to preload weights and kernels

Low RTF (< 1.0, slower than real-time)

• Increase blocksize — larger audio blocks reduce per-chunk overhead
• Disable unnecessary resampling — check post-processing pipeline in your TTS handler
• Profile GPU utilization — RTF < 1 often indicates compute-bound generation

STT Streaming Delays

Check smart_progressive_streaming.py for per-chunk inference logs. If Inference X.XX ms exceeds your target latency budget, consider:

• Reducing model precision (FP16 → INT8)
• Adjusting vad_sensitivity to trigger transcription less frequently

Key Source Files for Latency Analysis

Component	File Path	Primary Functions
Qwen3-TTS timing	src/speech_to_speech/TTS/qwen3_tts_handler.py	_stream() — TTFA and RTF logging
Pocket TTS timing	src/speech_to_speech/TTS/pocket_tts_handler.py	_stream() — first-audio measurement
STT streaming	src/speech_to_speech/STT/smart_progressive_streaming.py	smart_progressive_streaming(), smart_turn()
Session metrics	src/speech_to_speech/api/openai_realtime/service.py	GlobalUsageMetrics, RealtimeService.unregister()
TTS arguments	src/speech_to_speech/arguments_classes/chat_tts_arguments.py	streaming_chunk_size parameter

Summary

• TTFA and RTF logs in qwen3_tts_handler.py provide immediate visibility into TTS latency characteristics
• Per-chunk STT timing in smart_progressive_streaming.py reveals transcription bottlenecks during streaming
• GlobalUsageMetrics offers extensible infrastructure for session-wide latency aggregation
• Benchmark scripts (scripts/benchmark_tts.py, scripts/benchmark_stt.py) establish reproducible baselines
• Configuration tuning of streaming_chunk_size and blocksize directly impacts observed latency metrics

Frequently Asked Questions

How do I enable DEBUG-level logging to see all latency measurements?

Set LOG_LEVEL=DEBUG as an environment variable or configure logging.basicConfig(level=logging.DEBUG) before importing the speech-to-speech modules. This exposes per-chunk STT inference times and verbose TTS generation details.

What TTFA value indicates a properly optimized TTS setup?

TTFA below 200-300ms is achievable with warm models and appropriate streaming_chunk_size values. Values consistently above 500ms suggest model loading overhead or excessively large initial buffers.

Can I export latency metrics to Prometheus or another metrics system?

Yes. Extend the GlobalUsageMetrics class in src/speech_to_speech/api/openai_realtime/service.py to compute histograms from the per-handler logs, then expose them through your preferred metrics endpoint during RealtimeService.unregister() cleanup.

Why is my RTF greater than 1.0 but TTFA is acceptable?

RTF > 1 indicates that total generation time exceeds audio duration, causing audible gaps or stuttering despite quick first-chunk response. This typically stems from slow token generation or post-processing bottlenecks after the initial output. Increase blocksize or optimize the synthesis backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}