# How to Benchmark TTS Quality and Performance Metrics in Fish-Speech

> Benchmark TTS quality and performance with Fish-Speech evaluation scripts. Measure WER, MOS, RTF, and throughput using Seed-TTS and EmergentTTS-Eval for accurate results.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Fish-Speech provides built-in evaluation scripts in [`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py) and inference engine hooks to measure both subjective quality (WER, MOS) and objective performance (RTF, throughput) using official benchmark suites like Seed-TTS and EmergentTTS-Eval.**

The fishaudio/fish-speech repository ships with comprehensive benchmarking capabilities that let you objectively compare its text-to-speech quality against industry standards and measure inference speed on your specific hardware. Whether you are reproducing the published Seed-TTS WER scores or calculating real-time factors for production deployment, the codebase provides dedicated scripts and utility functions to benchmark TTS quality and performance metrics accurately.

## Quality Benchmarks

Fish-Speech reports results on four standardized evaluation sets that cover word-error rate, naturalness, and instruction following.

### Seed-TTS WER and Audio Turing Test

The **Seed-TTS WER** benchmark measures Word-Error-Rate on the public Seed-TTS evaluation set for Chinese and English. Lower scores indicate better intelligibility. The **Audio Turing Test** reports human preference for naturalness when comparing model-generated audio against ground-truth, expressed as a posterior mean. Both metrics are listed in the **Benchmark Results** table in [[`README.md`](https://github.com/fishaudio/fish-speech/blob/main/README.md)](https://github.com/fishaudio/fish-speech/blob/main/README.md).

### EmergentTTS-Eval and Fish Instruction Benchmark

**EmergentTTS-Eval** provides a multi-dimensional win-rate covering paralinguistics, questions, and syntactic complexity. The **Fish Instruction Benchmark** measures Task-Adequacy-Rate (TAR) for task-adapted TTS alongside a MOS-style quality score. These results are also documented in the README and localized versions in [`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md) and [`docs/zh/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/zh/index.md).

## Performance Benchmarks

Objective speed metrics are essential for production deployment. Fish-Speech tracks three key indicators measured on hardware such as the H200 GPU.

### Real-Time Factor (RTF)

The **Real-Time Factor (RTF)** is calculated as inference time divided by audio duration. An RTF below 1.0 indicates faster-than-real-time synthesis. According to the repository's benchmark data, Fish-Speech achieves an RTF of **0.195** on an H200 GPU, making it approximately 5× faster than real time.

### Throughput and Time-to-First-Audio

**Throughput** measures the number of acoustic tokens processed per second while maintaining RTF below 0.5, typically exceeding **3,000 tokens/s** on high-end hardware. **Time-to-first-audio** measures latency from request initiation to the first audio chunk being streamed, averaging approximately **100 ms**. These figures are reported in the **Benchmark Results** section of the README.

## Running Your Own Benchmarks

The repository includes executable scripts and Python APIs to reproduce these metrics on your own datasets and hardware.

### Measuring Semantic Loss with eval_in_context.py

The [`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py) script serves as a proxy for quality evaluation by computing semantic loss on the Seed-TTS test set. It loads a checkpoint, runs evaluation batches, and generates a `semantic_loss.png` visualization.

```bash
python -m tools.llama.eval_in_context

```

This command downloads the required checkpoint (`checkpoints/text2semantic-sft-medium-v1.1-4k.pth`) and test data, then plots per-frame semantic loss. The source implementation is located at [[`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py)](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py).

### Calculating RTF During Inference

To measure Real-Time Factor programmatically, wrap the `InferenceEngine` with a timer. The engine resides in [`fish_speech/inference_engine/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py).

```python
import time
from fish_speech.inference_engine import InferenceEngine
from fish_speech.utils.schema import ServeTTSRequest

engine = InferenceEngine.load_from_checkpoint("checkpoints/s2-pro.pth")
req = ServeTTSRequest(
    text="Hello world!",
    max_new_tokens=200,
    chunk_length=30,
    top_p=0.9,
    repetition_penalty=1.1,
    temperature=0.8,
    seed=42,
    use_memory_cache=True
)

start = time.time()
for result in engine.inference(req):
    if result.code == "final":
        break
elapsed = time.time() - start

audio_len_sec = len(result.audio[1]) / 44100  # assuming 44.1 kHz

rtf = elapsed / audio_len_sec
print(f"RTF = {rtf:.3f}")

```

This snippet reports the RTF for a single utterance. Run this on your target GPU to reproduce the **0.195** figure quoted in the README.

### Computing Throughput

Throughput is calculated by counting acoustic tokens processed over time. Use the same `InferenceEngine`, but accumulate `chunk_length` tokens for each segment returned.

```python
import time
from fish_speech.inference_engine import InferenceEngine
from fish_speech.utils.schema import ServeTTSRequest

engine = InferenceEngine.load_from_checkpoint("checkpoints/s2-pro.pth")
req = ServeTTSRequest(
    text="Long paragraph for throughput testing...",
    max_new_tokens=4000,
    chunk_length=30,
    top_p=0.9,
    repetition_penalty=1.1,
    temperature=0.8,
    seed=42,
    use_memory_cache=True
)

tokens_processed = 0
start = time.time()
for result in engine.inference(req):
    if result.code == "segment":
        tokens_processed += req.chunk_length
    if result.code == "final":
        break
elapsed = time.time() - start

throughput = tokens_processed / elapsed
print(f"Throughput: {throughput:.0f} tokens/s")

```

The reported throughput (exceeding **3,000 tokens/s**) matches the numbers in the README when run on appropriate hardware.

### Retrieving Training Metrics

After a training run, Lightning logs metrics such as WER. Use `get_metric_value` from [`fish_speech/utils/utils.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py) to extract specific values.

```python
from fish_speech.utils import get_metric_value

# `trainer` is the Lightning Trainer from fish_speech/train.py

metric_name = "wer_en"  # example metric key used in the trainer

wer = get_metric_value(trainer.callback_metrics, metric_name)
print(f"English WER: {wer:.2%}")

```

This utility safely handles missing metrics and is located at [[`fish_speech/utils/utils.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py)](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py).

## Key Implementation Files

Understanding the source locations helps when customizing benchmarks:

- **[`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py)** – Semantic loss evaluation and quality proxy scripts.
- **[`fish_speech/inference_engine/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py)** – Core `InferenceEngine` class for RTF and throughput measurement.
- **[`fish_speech/utils/utils.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py)** – Helper utilities including `get_metric_value` for training metrics.
- **[`fish_speech/train.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/train.py)** – Training entry point that logs WER and other quality metrics.
- **[`README.md`](https://github.com/fishaudio/fish-speech/blob/main/README.md)** – Central documentation of benchmark results and methodology.
- **[`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md)** and **[`docs/zh/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/zh/index.md)** – Localized benchmark tables and explanations.

## Summary

- Fish-Speech provides **built-in evaluation scripts** ([`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py)) to measure semantic loss as a proxy for synthesis quality.
- **Quality metrics** include Seed-TTS WER, Audio Turing Test scores, EmergentTTS-Eval win-rates, and Fish Instruction Benchmark TAR scores, all reported in the README.
- **Performance metrics** are measured via the `InferenceEngine` class, tracking Real-Time Factor (RTF ~0.195), throughput (>3,000 tokens/s), and time-to-first-audio (~100ms).
- Use `get_metric_value` from [`fish_speech/utils/utils.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py) to extract WER and other metrics from Lightning training logs.
- All benchmark implementations are open-source in `fishaudio/fish-speech`, allowing reproducible evaluation on custom hardware and datasets.

## Frequently Asked Questions

### How do I reproduce the Seed-TTS WER scores reported in the Fish-Speech README?

Run the semantic loss evaluation script located at [`tools/llama/eval_in_context.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py). This script loads the official checkpoint and runs inference on the Seed-TTS test set, calculating per-frame semantic loss which correlates with WER. Execute it with `python -m tools.llama.eval_in_context` and compare the generated `semantic_loss.png` against the published benchmarks.

### What is the difference between RTF and throughput in TTS benchmarking?

**Real-Time Factor (RTF)** measures the ratio of inference time to audio duration, indicating whether synthesis is faster than real-time (RTF < 1). **Throughput** measures the number of acoustic tokens processed per second, indicating raw computational efficiency. Fish-Speech achieves an RTF of approximately 0.195 and throughput exceeding 3,000 tokens per second on H200 GPUs, meaning it generates 5 seconds of audio in roughly 1 second while processing thousands of tokens continuously.

### Which hardware configuration was used for the official performance benchmarks?

According to the repository's benchmark data in [`README.md`](https://github.com/fishaudio/fish-speech/blob/main/README.md) and [`docs/en/index.md`](https://github.com/fishaudio/fish-speech/blob/main/docs/en/index.md), the official performance metrics (RTF ~0.195, throughput >3,000 tokens/s, time-to-first-audio ~100ms) were measured on an **H200 GPU**. You can reproduce these figures on your own hardware by wrapping the `InferenceEngine` with the timing code provided in the performance benchmarking examples.

### How do I extract WER metrics from a training run?

Use the `get_metric_value` utility function located in [`fish_speech/utils/utils.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py). After training with the Lightning Trainer in [`fish_speech/train.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/train.py), import the function and pass the trainer's `callback_metrics` dictionary along with the metric name (e.g., `"wer_en"` for English Word-Error-Rate). This safely retrieves the logged value for comparison against the Seed-TTS benchmark targets.