How to Benchmark TTS Quality and Performance Metrics in Fish-Speech

Fish-Speech provides built-in evaluation scripts in tools/llama/eval_in_context.py and inference engine hooks to measure both subjective quality (WER, MOS) and objective performance (RTF, throughput) using official benchmark suites like Seed-TTS and EmergentTTS-Eval.

The fishaudio/fish-speech repository ships with comprehensive benchmarking capabilities that let you objectively compare its text-to-speech quality against industry standards and measure inference speed on your specific hardware. Whether you are reproducing the published Seed-TTS WER scores or calculating real-time factors for production deployment, the codebase provides dedicated scripts and utility functions to benchmark TTS quality and performance metrics accurately.

Quality Benchmarks

Fish-Speech reports results on four standardized evaluation sets that cover word-error rate, naturalness, and instruction following.

Seed-TTS WER and Audio Turing Test

The Seed-TTS WER benchmark measures Word-Error-Rate on the public Seed-TTS evaluation set for Chinese and English. Lower scores indicate better intelligibility. The Audio Turing Test reports human preference for naturalness when comparing model-generated audio against ground-truth, expressed as a posterior mean. Both metrics are listed in the Benchmark Results table in [README.md](https://github.com/fishaudio/fish-speech/blob/main/README.md).

EmergentTTS-Eval and Fish Instruction Benchmark

EmergentTTS-Eval provides a multi-dimensional win-rate covering paralinguistics, questions, and syntactic complexity. The Fish Instruction Benchmark measures Task-Adequacy-Rate (TAR) for task-adapted TTS alongside a MOS-style quality score. These results are also documented in the README and localized versions in docs/en/index.md and docs/zh/index.md.

Performance Benchmarks

Objective speed metrics are essential for production deployment. Fish-Speech tracks three key indicators measured on hardware such as the H200 GPU.

Real-Time Factor (RTF)

The Real-Time Factor (RTF) is calculated as inference time divided by audio duration. An RTF below 1.0 indicates faster-than-real-time synthesis. According to the repository's benchmark data, Fish-Speech achieves an RTF of 0.195 on an H200 GPU, making it approximately 5× faster than real time.

Throughput and Time-to-First-Audio

Throughput measures the number of acoustic tokens processed per second while maintaining RTF below 0.5, typically exceeding 3,000 tokens/s on high-end hardware. Time-to-first-audio measures latency from request initiation to the first audio chunk being streamed, averaging approximately 100 ms. These figures are reported in the Benchmark Results section of the README.

Running Your Own Benchmarks

The repository includes executable scripts and Python APIs to reproduce these metrics on your own datasets and hardware.

Measuring Semantic Loss with eval_in_context.py

The tools/llama/eval_in_context.py script serves as a proxy for quality evaluation by computing semantic loss on the Seed-TTS test set. It loads a checkpoint, runs evaluation batches, and generates a semantic_loss.png visualization.

python -m tools.llama.eval_in_context

This command downloads the required checkpoint (checkpoints/text2semantic-sft-medium-v1.1-4k.pth) and test data, then plots per-frame semantic loss. The source implementation is located at [tools/llama/eval_in_context.py](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/eval_in_context.py).

Calculating RTF During Inference

To measure Real-Time Factor programmatically, wrap the InferenceEngine with a timer. The engine resides in fish_speech/inference_engine/__init__.py.

import time
from fish_speech.inference_engine import InferenceEngine
from fish_speech.utils.schema import ServeTTSRequest

engine = InferenceEngine.load_from_checkpoint("checkpoints/s2-pro.pth")
req = ServeTTSRequest(
    text="Hello world!",
    max_new_tokens=200,
    chunk_length=30,
    top_p=0.9,
    repetition_penalty=1.1,
    temperature=0.8,
    seed=42,
    use_memory_cache=True
)

start = time.time()
for result in engine.inference(req):
    if result.code == "final":
        break
elapsed = time.time() - start

audio_len_sec = len(result.audio[1]) / 44100  # assuming 44.1 kHz

rtf = elapsed / audio_len_sec
print(f"RTF = {rtf:.3f}")

This snippet reports the RTF for a single utterance. Run this on your target GPU to reproduce the 0.195 figure quoted in the README.

Computing Throughput

Throughput is calculated by counting acoustic tokens processed over time. Use the same InferenceEngine, but accumulate chunk_length tokens for each segment returned.

import time
from fish_speech.inference_engine import InferenceEngine
from fish_speech.utils.schema import ServeTTSRequest

engine = InferenceEngine.load_from_checkpoint("checkpoints/s2-pro.pth")
req = ServeTTSRequest(
    text="Long paragraph for throughput testing...",
    max_new_tokens=4000,
    chunk_length=30,
    top_p=0.9,
    repetition_penalty=1.1,
    temperature=0.8,
    seed=42,
    use_memory_cache=True
)

tokens_processed = 0
start = time.time()
for result in engine.inference(req):
    if result.code == "segment":
        tokens_processed += req.chunk_length
    if result.code == "final":
        break
elapsed = time.time() - start

throughput = tokens_processed / elapsed
print(f"Throughput: {throughput:.0f} tokens/s")

The reported throughput (exceeding 3,000 tokens/s) matches the numbers in the README when run on appropriate hardware.

Retrieving Training Metrics

After a training run, Lightning logs metrics such as WER. Use get_metric_value from fish_speech/utils/utils.py to extract specific values.

from fish_speech.utils import get_metric_value

# `trainer` is the Lightning Trainer from fish_speech/train.py

metric_name = "wer_en"  # example metric key used in the trainer

wer = get_metric_value(trainer.callback_metrics, metric_name)
print(f"English WER: {wer:.2%}")

This utility safely handles missing metrics and is located at [fish_speech/utils/utils.py](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/utils/utils.py).

Key Implementation Files

Understanding the source locations helps when customizing benchmarks:

• tools/llama/eval_in_context.py – Semantic loss evaluation and quality proxy scripts.
• fish_speech/inference_engine/__init__.py – Core InferenceEngine class for RTF and throughput measurement.
• fish_speech/utils/utils.py – Helper utilities including get_metric_value for training metrics.
• fish_speech/train.py – Training entry point that logs WER and other quality metrics.
• README.md – Central documentation of benchmark results and methodology.
• docs/en/index.md and docs/zh/index.md – Localized benchmark tables and explanations.

Summary

• Fish-Speech provides built-in evaluation scripts (tools/llama/eval_in_context.py) to measure semantic loss as a proxy for synthesis quality.
• Quality metrics include Seed-TTS WER, Audio Turing Test scores, EmergentTTS-Eval win-rates, and Fish Instruction Benchmark TAR scores, all reported in the README.
• Performance metrics are measured via the InferenceEngine class, tracking Real-Time Factor (RTF ~0.195), throughput (>3,000 tokens/s), and time-to-first-audio (~100ms).
• Use get_metric_value from fish_speech/utils/utils.py to extract WER and other metrics from Lightning training logs.
• All benchmark implementations are open-source in fishaudio/fish-speech, allowing reproducible evaluation on custom hardware and datasets.

Frequently Asked Questions

How do I reproduce the Seed-TTS WER scores reported in the Fish-Speech README?

Run the semantic loss evaluation script located at tools/llama/eval_in_context.py. This script loads the official checkpoint and runs inference on the Seed-TTS test set, calculating per-frame semantic loss which correlates with WER. Execute it with python -m tools.llama.eval_in_context and compare the generated semantic_loss.png against the published benchmarks.

What is the difference between RTF and throughput in TTS benchmarking?

Real-Time Factor (RTF) measures the ratio of inference time to audio duration, indicating whether synthesis is faster than real-time (RTF < 1). Throughput measures the number of acoustic tokens processed per second, indicating raw computational efficiency. Fish-Speech achieves an RTF of approximately 0.195 and throughput exceeding 3,000 tokens per second on H200 GPUs, meaning it generates 5 seconds of audio in roughly 1 second while processing thousands of tokens continuously.

Which hardware configuration was used for the official performance benchmarks?

According to the repository's benchmark data in README.md and docs/en/index.md, the official performance metrics (RTF ~0.195, throughput >3,000 tokens/s, time-to-first-audio ~100ms) were measured on an H200 GPU. You can reproduce these figures on your own hardware by wrapping the InferenceEngine with the timing code provided in the performance benchmarking examples.

How do I extract WER metrics from a training run?

Use the get_metric_value utility function located in fish_speech/utils/utils.py. After training with the Lightning Trainer in fish_speech/train.py, import the function and pass the trainer's callback_metrics dictionary along with the metric name (e.g., "wer_en" for English Word-Error-Rate). This safely retrieves the logged value for comparison against the Seed-TTS benchmark targets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}