How to Optimize Inference Latency with SGLang Streaming in Fish-Speech

Enable SGLang streaming in Fish-Speech to achieve approximately 100 ms time-to-first-audio and a real-time factor (RTF) below 0.2 on modern GPUs.

Fish-Speech is an open-source text-to-speech system from fishaudio/fish-speech that uses a dual-autoregressive (Dual-AR) transformer architecture. Because the model generates semantic tokens through a standard LLaMA-style decoder, it can leverage SGLang, an LLM-native serving stack, to drastically reduce inference latency. This guide explains how to configure and deploy SGLang streaming optimizations using the exact implementation paths found in the repository.

Why SGLang Acceleration Works for Fish-Speech

Fish-Speech’s inference pipeline treats audio generation as a token-decoding problem. The TTSInferenceEngine calls an internal LLaMA request (send_Llama_request), making the workload identical to standard large-language-model inference. SGLang exploits this by applying kernel-level optimizations that minimize per-request overhead and maximize GPU utilization.

Key SGLang Optimizations for Low-Latency Streaming

Continuous Batching and Paged KV-Cache

Continuous batching packs multiple incoming requests into a single CUDA forward pass, eliminating the latency of per-request kernel launches. Paged KV-cache stores only active attention states in fixed-size pages, allowing Fish-Speech to handle long reference audio prompts without exhausting GPU memory. Together, these enable high-throughput serving of multiple concurrent TTS streams.

CUDA Graph Replay and Radix-Attention Prefix Caching

CUDA-graph replay records the kernel launch sequence once and re-executes it without driver overhead, which is critical for short-generation scenarios common in conversational TTS. Radix-Attention prefix caching shares attention computations across requests that use identical speaker references. If multiple users synthesize speech with the same voice prompt, the cached prefix eliminates redundant computation, cutting time-to-first-token dramatically.

Streaming Output Architecture

When req.streaming=True, the inference engine yields audio chunks as they are decoded rather than waiting for the full sequence. In fish_speech/inference_engine/__init__.py, the engine first yields a WAV header (code="header") at lines 73-84, then yields audio segments (code="segment") at lines 111-117, and finally a final block (code="final"). The server in tools/server/views.py (lines 147-156) wraps these chunks in an HTTP StreamResponse, allowing clients to begin playback within milliseconds.

Implementation Guide: Enabling SGLang Streaming

Step 1 – Deploy the SGLang-Omni Server

The Fish-Speech repository references the SGLang-Omni server for production deployment (see README.md lines 65-66). Start the server to expose the optimized LLaMA endpoint:

cd /path/to/sglang-omni
python -m sglang_omni.server \
    --model-dir ./models/fishaudio_s2_pro \
    --port 8081

This initializes the continuous batching, paged KV-cache, and CUDA-graph engines required for low-latency inference.

Step 2 – Launch Fish-Speech with CUDA Kernel Fusion

Start the Fish-Speech API with the --compile flag to enable CUDA kernel fusion inside the LLaMA decoder. This works in conjunction with SGLang’s optimizations:

python -m tools.run_webui \
    --mode tts \
    --llama-checkpoint-path checkpoints/s2-pro \
    --decoder-checkpoint-path checkpoints/s2-pro/codec.pth \
    --device cuda \
    --compile

The --compile flag triggers graph-based execution paths that minimize Python overhead during token generation.

Step 3 – Stream from the Command Line

Use the built-in API client to test streaming latency. The client in tools/api_client.py supports the --streaming flag (lines 167-189) to enable HTTP chunked transfer and real-time playback:

python -m tools.api_client \
    --url http://127.0.0.1:8080/v1/tts \
    --text "Low latency streaming with SGLang acceleration." \
    --format wav \
    --streaming \
    --output low_latency_demo

Audio playback begins as soon as the first segment arrives, rather than waiting for the full file.

Step 4 – Integrate Streaming in Python

For production applications, use httpx to consume the streaming endpoint with MessagePack serialization. The server accepts ServeTTSRequest objects and returns raw PCM chunks:

import httpx
import ormsgack
from fish_speech.utils.schema import ServeTTSRequest

req = ServeTTSRequest(
    text="Streaming demo with SGLang.",
    references=[],
    reference_id="",
    format="wav",
    max_new_tokens=256,
    chunk_length=0,
    top_p=0.9,
    repetition_penalty=1.0,
    temperature=0.7,
    streaming=True,
    use_memory_cache=False,
    seed=42,
)

packed = ormsgack.packb(req, option=ormsgack.OPT_SERIALIZE_PYDANTIC)

with httpx.stream(
    "POST",
    "http://127.0.0.1:8080/v1/tts?format=msgpack",
    content=packed,
    headers={"content-type": "application/msgpack"},
    timeout=None,
) as resp:
    for chunk in resp.iter_bytes():
        # Process raw PCM-16 audio chunk immediately

        process_audio(chunk)  # Replace with your playback logic

This pattern achieves time-to-first-audio of approximately 100 ms on an NVIDIA H200 GPU by eliminating buffering delays.

Verifying Latency Reduction

Measure end-to-end latency using the time command with the streaming client:

time python -m tools.api_client \
    --url http://127.0.0.1:8080/v1/tts \
    --text "Latency benchmark" \
    --streaming

On optimized hardware, expect:

• Time-to-first-audio: ~100 ms
• Real-time factor (RTF): < 0.2
• Throughput: > 3,000 audio tokens/second

Summary

• SGLang streaming reduces Fish-Speech inference latency by treating the Dual-AR transformer as a standard LLM and applying kernel-level optimizations.
• Key techniques include continuous batching, paged KV-cache, CUDA-graph replay, and Radix-Attention prefix caching.
• Implementation requires deploying the SGLang-Omni server, launching Fish-Speech with --compile, and enabling streaming=True in requests.
• Performance targets achieve ~100 ms time-to-first-audio and sub-0.2 RTF on modern GPUs by streaming raw PCM chunks via InferenceResult codes (header, segment, final).

Frequently Asked Questions

What is the expected time-to-first-audio with SGLang streaming?

With SGLang streaming enabled on an NVIDIA H200 GPU, Fish-Speech achieves approximately 100 milliseconds to the first audio chunk. This is accomplished by bypassing full-file generation and immediately streaming PCM segments as they are decoded by the LLaMA-style transformer.

How does the Dual-AR architecture benefit from LLM serving optimizations?

Fish-Speech uses a dual-autoregressive design where a semantic LLaMA decoder generates tokens that are then converted to audio. Because the token-generation loop is mathematically identical to standard large-language-model inference, it inherits optimizations like continuous batching, paged KV-cache, and CUDA-graph replay from SGLang without architectural changes.

Can I use SGLang streaming with the WebUI or only via API?

SGLang streaming is primarily exposed through the API layer (tools/server/views.py) and the command-line client (tools/api_client.py). While the WebUI (tools/run_webui.py) supports the --compile flag for kernel fusion, real-time streaming playback requires consuming the chunked HTTP response programmatically or via the provided CLI client with the --streaming flag.

What hardware requirements are needed for sub-200ms RTF?

Achieving a real-time factor (RTF) below 0.2 and throughput exceeding 3,000 audio tokens per second requires a modern NVIDIA GPU such as the H200 or H100. The SGLang optimizations reduce CPU overhead and kernel launch latency, but saturating the continuous batching and paged attention mechanisms still demands high memory bandwidth and compute capacity typical of data-center GPUs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}