# How to Benchmark and Profile the Speech-to-Speech Pipeline in Hugging Face's S2S Library

> Benchmark and profile the Hugging Face speech-to-speech pipeline using built-in scripts and Python profilers. Measure end-to-end latency and analyze TTS/STT performance.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: performance
- Published: 2026-08-05

---

**The speech-to-speech library provides built-in benchmark scripts for TTS and STT components, plus a modular `SpeechToSpeechPipeline` class that enables end-to-end latency measurement and integration with standard Python profilers like `cProfile`, `torch.profiler`, and `memory_profiler`.**

This guide walks through the complete workflow for measuring performance in the [huggingface/speech-to-speech](https://github.com/huggingface/speech-to-speech) repository. Whether you need to optimize a single stage or the full STT → LLM → TTS pipeline, the codebase offers purpose-built tools and clear extension points.

## Benchmark the TTS Stage with [`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py)

The [`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py) script provides standardized measurements for any text-to-speech handler. It supports handlers like `qwen3_tts_handler`, `pocket_tts_handler`, and others defined in the repository.

The script implements a rigorous measurement protocol:

- Records wall-clock time around each `generate()` call
- Discards configurable warm-up runs to exclude JIT compilation overhead
- Reports per-run statistics: mean, median, and 95th-percentile latency
- Calculates **real-time factor (RTF)** by comparing generation time against total audio duration produced

Run it from the command line with custom parameters:

```bash
python scripts/benchmark_tts.py \
  --model "Qwen/Qwen3-Text-To-Audio" \
  --batch-size 1 \
  --reps 50

```

The RTF metric deserves special attention. An RTF of 0.5 means the system generates audio twice as fast as real-time playback. Values above 1.0 indicate slower-than-real-time generation, which creates unacceptable latency for interactive applications.

**Source**: [[`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py)](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py)

## Benchmark the STT Stage with [`benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_stt.py)

The [`scripts/benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py) script mirrors the TTS approach for automatic speech recognition handlers including Whisper, Faster-Whisper, and Moonshine variants.

Key measurements include:

- **Latency**: Time from audio input to text completion
- **Throughput**: Words-per-second processed during transcription

The script feeds a fixed audio clip repeatedly through the `transcribe` method, eliminating input variability that could distort comparisons between model configurations.

**Source**: [[`scripts/benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py)](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py)

## Measure End-to-End Speech-to-Speech Pipeline Latency

The `SpeechToSpeechPipeline` class in [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) orchestrates the complete STT → LLM → TTS flow. Its modular design lets you benchmark the integrated system or isolate specific bottlenecks.

Basic timing implementation:

```python
import time
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline
from speech_to_speech.arguments_classes.whisper_stt_arguments import WhisperSttArguments
from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TtsArguments
from speech_to_speech.arguments_classes.language_model_arguments import LanguageModelArguments

# Configure handlers

stt_args = WhisperSttArguments(model="openai/whisper-base")
lm_args = LanguageModelArguments(model="meta-llama/Meta-Llama-3-8B-Instruct")
tts_args = Qwen3TtsArguments(model="Qwen/Qwen3-Text-To-Audio")

pipeline = SpeechToSpeechPipeline(
    stt_args=stt_args,
    lm_args=lm_args,
    tts_args=tts_args,
)

# Warm-up run

pipeline.run("Warm-up phrase to load models and compile kernels.")

# Benchmark

reps = 10
latencies = []
for _ in range(reps):
    start = time.perf_counter()
    output = pipeline.run("Your test utterance here.")
    end = time.perf_counter()
    latencies.append(end - start)

print(f"Mean latency: {sum(latencies)/reps:.3f}s")

```

The `pipeline.run()` method accepts either audio input (triggering full STT → LLM → TTS flow) or text input (shortcutting to LLM → TTS). This flexibility supports both complete pipeline benchmarks and partial-path measurements.

**Source**: [[`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py)

## Profile with Standard Python and PyTorch Tools

The speech-to-speech pipeline integrates cleanly with industry-standard profilers. Choose your tool based on the optimization target:

### CPU Hotspot Analysis with `cProfile`

Identify Python-level bottlenecks in handler logic or pipeline orchestration:

```python
import cProfile
import pstats
import io

pr = cProfile.Profile()
pr.enable()

pipeline.run("Test input for profiling")

pr.disable()
s = io.StringIO()
ps = pstats.Stats(pr, stream=s).sort_stats('cumtime')
ps.print_stats(20)
print(s.getvalue())

```

Visualize results with `snakeviz` for interactive exploration of call graphs.

### GPU Kernel Timeline with `torch.profiler`

Essential for CUDA-accelerated models like Whisper or Llama-based LLMs:

```python
from torch.profiler import profile, record_function, ProfilerActivity

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    record_shapes=True,
    with_stack=True
) as prof:
    with record_function("full_pipeline"):
        pipeline.run("Profile this utterance.")

print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
prof.export_chrome_trace("pipeline_trace.json")

```

The Chrome trace export loads into `chrome://tracing` or [Perfetto UI](https://ui.perfetto.dev) for nanosecond-accurate visualization of kernel scheduling and memory transfers.

### Memory Profiling with `memory_profiler`

Track heap growth and detect leaks in long-running pipeline instances:

```python
from memory_profiler import profile

@profile
def benchmark_run():
    for i in range(100):
        pipeline.run(f"Iteration {i}")

benchmark_run()

```

### Real-Time Factor Calculation

The [`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py) script exposes `total_duration` for RTF computation. Extend this to full pipeline measurements:

```python
total_audio_duration = sum(o.duration for o in outputs)
rtf = total_audio_duration / total_generation_time

```

RTF < 1.0 guarantees the pipeline can sustain real-time conversation without buffering delays.

## Reproducible Benchmarking Best Practices

Follow these guidelines to generate comparable, defensible performance numbers:

- **Warm-up protocols**: Execute 3-5 dry runs before measurement to stabilize GPU memory allocation, kernel caching, and model compilation
- **Fixed inputs**: Use identical audio clips or text prompts across all test configurations
- **Environment locking**: Pin `speech-to-speech` version, Python version, CUDA driver, and model weights with SHA checksums
- **Thread control**: Set `OMP_NUM_THREADS=1` and `MKL_NUM_THREADS=1` for single-threaded baselines; scale intentionally for parallel throughput tests
- **Thermal stability**: Allow GPU to reach steady-state temperature before recording; thermal throttling can introduce 10-20% variance

## Summary

- The speech-to-speech repository ships with [`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py) and [`scripts/benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py) for component-level latency and RTF measurement
- The `SpeechToSpeechPipeline` class in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) enables end-to-end benchmarking with standard Python timing utilities
- Integration with `cProfile`, `torch.profiler`, and `memory_profiler` provides deep visibility into CPU, GPU, and memory behavior
- Real-time factor (RTF) serves as the critical metric for interactive speech system viability

## Frequently Asked Questions

### Does the speech-to-speech library include automated benchmarking workflows?

No automated CI benchmarks are included in the repository. The [`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py) and [`scripts/benchmark_stt.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_stt.py) utilities are designed for manual execution with configurable parameters. You can integrate these into your own CI pipeline by wrapping the command-line interfaces and asserting on latency thresholds or RTF bounds.

### Which profiling tool should I use for GPU-accelerated TTS models?

Use `torch.profiler` with `ProfilerActivity.CUDA` enabled. This captures kernel execution times, memory bandwidth utilization, and operator-level breakdowns. For models running on CPU-only inference, `cProfile` with `cumtime` sorting identifies Python hotspots more efficiently without CUDA overhead.

### How do I measure streaming latency versus total generation time?

The current benchmark scripts measure total wall-clock time for complete generation. For streaming latency (time-to-first-audio), instrument the `generate()` method in your specific TTS handler to emit timestamps at the first audio chunk production. The modular handler architecture in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) supports this instrumentation without core pipeline modifications.

### What RTF value indicates real-time capable performance?

An RTF below 1.0 guarantees real-time capability. For conversational AI, target RTF ≤ 0.3 to accommodate network overhead and maintain natural turn-taking. The [`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py) script prints RTF automatically when the handler exposes audio duration metadata.