How to Benchmark and Profile the Speech-to-Speech Pipeline in Hugging Face's S2S Library

The speech-to-speech library provides built-in benchmark scripts for TTS and STT components, plus a modular SpeechToSpeechPipeline class that enables end-to-end latency measurement and integration with standard Python profilers like cProfile, torch.profiler, and memory_profiler.

This guide walks through the complete workflow for measuring performance in the huggingface/speech-to-speech repository. Whether you need to optimize a single stage or the full STT → LLM → TTS pipeline, the codebase offers purpose-built tools and clear extension points.

Benchmark the TTS Stage with benchmark_tts.py

The scripts/benchmark_tts.py script provides standardized measurements for any text-to-speech handler. It supports handlers like qwen3_tts_handler, pocket_tts_handler, and others defined in the repository.

The script implements a rigorous measurement protocol:

• Records wall-clock time around each generate() call
• Discards configurable warm-up runs to exclude JIT compilation overhead
• Reports per-run statistics: mean, median, and 95th-percentile latency
• Calculates real-time factor (RTF) by comparing generation time against total audio duration produced

Run it from the command line with custom parameters:

python scripts/benchmark_tts.py \
  --model "Qwen/Qwen3-Text-To-Audio" \
  --batch-size 1 \
  --reps 50

The RTF metric deserves special attention. An RTF of 0.5 means the system generates audio twice as fast as real-time playback. Values above 1.0 indicate slower-than-real-time generation, which creates unacceptable latency for interactive applications.

Source: [scripts/benchmark_tts.py](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py)

Benchmark the STT Stage with benchmark_stt.py

The scripts/benchmark_stt.py script mirrors the TTS approach for automatic speech recognition handlers including Whisper, Faster-Whisper, and Moonshine variants.

Key measurements include:

• Latency: Time from audio input to text completion
• Throughput: Words-per-second processed during transcription

The script feeds a fixed audio clip repeatedly through the transcribe method, eliminating input variability that could distort comparisons between model configurations.

Source: [scripts/benchmark_stt.py](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py)

Measure End-to-End Speech-to-Speech Pipeline Latency

The SpeechToSpeechPipeline class in src/speech_to_speech/s2s_pipeline.py orchestrates the complete STT → LLM → TTS flow. Its modular design lets you benchmark the integrated system or isolate specific bottlenecks.

Basic timing implementation:

import time
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.whisper_stt_arguments import WhisperSttArguments
from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TtsArguments
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelArguments

# Configure handlers

stt_args = WhisperSttArguments(model="openai/whisper-base")
lm_args = LanguageModelArguments(model="meta-llama/Meta-Llama-3-8B-Instruct")
tts_args = Qwen3TtsArguments(model="Qwen/Qwen3-Text-To-Audio")

pipeline = SpeechToSpeechPipeline(
    stt_args=stt_args,
    lm_args=lm_args,
    tts_args=tts_args,
)

# Warm-up run

pipeline.run("Warm-up phrase to load models and compile kernels.")

# Benchmark

reps = 10
latencies = []
for _ in range(reps):
    start = time.perf_counter()
    output = pipeline.run("Your test utterance here.")
    end = time.perf_counter()
    latencies.append(end - start)

print(f"Mean latency: {sum(latencies)/reps:.3f}s")

The pipeline.run() method accepts either audio input (triggering full STT → LLM → TTS flow) or text input (shortcutting to LLM → TTS). This flexibility supports both complete pipeline benchmarks and partial-path measurements.

Source: [src/speech_to_speech/s2s_pipeline.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)

Profile with Standard Python and PyTorch Tools

The speech-to-speech pipeline integrates cleanly with industry-standard profilers. Choose your tool based on the optimization target:

CPU Hotspot Analysis with cProfile

Identify Python-level bottlenecks in handler logic or pipeline orchestration:

import cProfile
import pstats
import io

pr = cProfile.Profile()
pr.enable()

pipeline.run("Test input for profiling")

pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('cumtime')
ps.print_stats(20)
print(s.getvalue())

Visualize results with snakeviz for interactive exploration of call graphs.

GPU Kernel Timeline with torch.profiler

Essential for CUDA-accelerated models like Whisper or Llama-based LLMs:

from torch.profiler import profile, record_function, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    with_stack=True
) as prof:
    with record_function("full_pipeline"):
        pipeline.run("Profile this utterance.")

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
prof.export_chrome_trace("pipeline_trace.json")

The Chrome trace export loads into chrome://tracing or Perfetto UI for nanosecond-accurate visualization of kernel scheduling and memory transfers.

Memory Profiling with memory_profiler

Track heap growth and detect leaks in long-running pipeline instances:

from memory_profiler import profile

@profile
def benchmark_run():
    for i in range(100):
        pipeline.run(f"Iteration {i}")

benchmark_run()

Real-Time Factor Calculation

The benchmark_tts.py script exposes total_duration for RTF computation. Extend this to full pipeline measurements:

total_audio_duration = sum(o.duration for o in outputs)
rtf = total_audio_duration / total_generation_time

RTF < 1.0 guarantees the pipeline can sustain real-time conversation without buffering delays.

Reproducible Benchmarking Best Practices

Follow these guidelines to generate comparable, defensible performance numbers:

• Warm-up protocols: Execute 3-5 dry runs before measurement to stabilize GPU memory allocation, kernel caching, and model compilation
• Fixed inputs: Use identical audio clips or text prompts across all test configurations
• Environment locking: Pin speech-to-speech version, Python version, CUDA driver, and model weights with SHA checksums
• Thread control: Set OMP_NUM_THREADS=1 and MKL_NUM_THREADS=1 for single-threaded baselines; scale intentionally for parallel throughput tests
• Thermal stability: Allow GPU to reach steady-state temperature before recording; thermal throttling can introduce 10-20% variance

Summary

• The speech-to-speech repository ships with scripts/benchmark_tts.py and scripts/benchmark_stt.py for component-level latency and RTF measurement
• The SpeechToSpeechPipeline class in s2s_pipeline.py enables end-to-end benchmarking with standard Python timing utilities
• Integration with cProfile, torch.profiler, and memory_profiler provides deep visibility into CPU, GPU, and memory behavior
• Real-time factor (RTF) serves as the critical metric for interactive speech system viability

Frequently Asked Questions

Does the speech-to-speech library include automated benchmarking workflows?

No automated CI benchmarks are included in the repository. The scripts/benchmark_tts.py and scripts/benchmark_stt.py utilities are designed for manual execution with configurable parameters. You can integrate these into your own CI pipeline by wrapping the command-line interfaces and asserting on latency thresholds or RTF bounds.

Which profiling tool should I use for GPU-accelerated TTS models?

Use torch.profiler with ProfilerActivity.CUDA enabled. This captures kernel execution times, memory bandwidth utilization, and operator-level breakdowns. For models running on CPU-only inference, cProfile with cumtime sorting identifies Python hotspots more efficiently without CUDA overhead.

How do I measure streaming latency versus total generation time?

The current benchmark scripts measure total wall-clock time for complete generation. For streaming latency (time-to-first-audio), instrument the generate() method in your specific TTS handler to emit timestamps at the first audio chunk production. The modular handler architecture in s2s_pipeline.py supports this instrumentation without core pipeline modifications.

What RTF value indicates real-time capable performance?

An RTF below 1.0 guarantees real-time capability. For conversational AI, target RTF ≤ 0.3 to accommodate network overhead and maintain natural turn-taking. The benchmark_tts.py script prints RTF automatically when the handler exposes audio duration metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}