# How to Optimize Inference Latency with SGLang Streaming in Fish-Speech

> Optimize inference latency with SGLang streaming in Fish-Speech. Achieve 100ms time-to-first-audio and RTF below 0.2 on GPUs. Boost your speech AI performance today.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Enable SGLang streaming in Fish-Speech to achieve approximately 100 ms time-to-first-audio and a real-time factor (RTF) below 0.2 on modern GPUs.**

Fish-Speech is an open-source text-to-speech system from `fishaudio/fish-speech` that uses a **dual-autoregressive (Dual-AR)** transformer architecture. Because the model generates semantic tokens through a standard LLaMA-style decoder, it can leverage **SGLang**, an LLM-native serving stack, to drastically reduce inference latency. This guide explains how to configure and deploy SGLang streaming optimizations using the exact implementation paths found in the repository.

## Why SGLang Acceleration Works for Fish-Speech

Fish-Speech’s inference pipeline treats audio generation as a token-decoding problem. The **TTSInferenceEngine** calls an internal LLaMA request (`send_Llama_request`), making the workload identical to standard large-language-model inference. SGLang exploits this by applying kernel-level optimizations that minimize per-request overhead and maximize GPU utilization.

## Key SGLang Optimizations for Low-Latency Streaming

### Continuous Batching and Paged KV-Cache

**Continuous batching** packs multiple incoming requests into a single CUDA forward pass, eliminating the latency of per-request kernel launches. **Paged KV-cache** stores only active attention states in fixed-size pages, allowing Fish-Speech to handle long reference audio prompts without exhausting GPU memory. Together, these enable high-throughput serving of multiple concurrent TTS streams.

### CUDA Graph Replay and Radix-Attention Prefix Caching

**CUDA-graph replay** records the kernel launch sequence once and re-executes it without driver overhead, which is critical for short-generation scenarios common in conversational TTS. **Radix-Attention prefix caching** shares attention computations across requests that use identical speaker references. If multiple users synthesize speech with the same voice prompt, the cached prefix eliminates redundant computation, cutting time-to-first-token dramatically.

### Streaming Output Architecture

When `req.streaming=True`, the inference engine yields audio chunks as they are decoded rather than waiting for the full sequence. In [`fish_speech/inference_engine/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/__init__.py), the engine first yields a **WAV header** (`code="header"`) at lines 73-84, then yields audio **segments** (`code="segment"`) at lines 111-117, and finally a **final block** (`code="final"`). The server in [`tools/server/views.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py) (lines 147-156) wraps these chunks in an HTTP `StreamResponse`, allowing clients to begin playback within milliseconds.

## Implementation Guide: Enabling SGLang Streaming

### Step 1 – Deploy the SGLang-Omni Server

The Fish-Speech repository references the SGLang-Omni server for production deployment (see [`README.md`](https://github.com/fishaudio/fish-speech/blob/main/README.md) lines 65-66). Start the server to expose the optimized LLaMA endpoint:

```bash
cd /path/to/sglang-omni
python -m sglang_omni.server \
    --model-dir ./models/fishaudio_s2_pro \
    --port 8081

```

This initializes the continuous batching, paged KV-cache, and CUDA-graph engines required for low-latency inference.

### Step 2 – Launch Fish-Speech with CUDA Kernel Fusion

Start the Fish-Speech API with the `--compile` flag to enable CUDA kernel fusion inside the LLaMA decoder. This works in conjunction with SGLang’s optimizations:

```bash
python -m tools.run_webui \
    --mode tts \
    --llama-checkpoint-path checkpoints/s2-pro \
    --decoder-checkpoint-path checkpoints/s2-pro/codec.pth \
    --device cuda \
    --compile

```

The `--compile` flag triggers graph-based execution paths that minimize Python overhead during token generation.

### Step 3 – Stream from the Command Line

Use the built-in API client to test streaming latency. The client in [`tools/api_client.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/api_client.py) supports the `--streaming` flag (lines 167-189) to enable HTTP chunked transfer and real-time playback:

```bash
python -m tools.api_client \
    --url http://127.0.0.1:8080/v1/tts \
    --text "Low latency streaming with SGLang acceleration." \
    --format wav \
    --streaming \
    --output low_latency_demo

```

Audio playback begins as soon as the first segment arrives, rather than waiting for the full file.

### Step 4 – Integrate Streaming in Python

For production applications, use `httpx` to consume the streaming endpoint with MessagePack serialization. The server accepts `ServeTTSRequest` objects and returns raw PCM chunks:

```python
import httpx
import ormsgack
from fish_speech.utils.schema import ServeTTSRequest

req = ServeTTSRequest(
    text="Streaming demo with SGLang.",
    references=[],
    reference_id="",
    format="wav",
    max_new_tokens=256,
    chunk_length=0,
    top_p=0.9,
    repetition_penalty=1.0,
    temperature=0.7,
    streaming=True,
    use_memory_cache=False,
    seed=42,
)

packed = ormsgack.packb(req, option=ormsgack.OPT_SERIALIZE_PYDANTIC)

with httpx.stream(
    "POST",
    "http://127.0.0.1:8080/v1/tts?format=msgpack",
    content=packed,
    headers={"content-type": "application/msgpack"},
    timeout=None,
) as resp:
    for chunk in resp.iter_bytes():
        # Process raw PCM-16 audio chunk immediately

        process_audio(chunk)  # Replace with your playback logic

```

This pattern achieves **time-to-first-audio** of approximately 100 ms on an NVIDIA H200 GPU by eliminating buffering delays.

## Verifying Latency Reduction

Measure end-to-end latency using the `time` command with the streaming client:

```bash
time python -m tools.api_client \
    --url http://127.0.0.1:8080/v1/tts \
    --text "Latency benchmark" \
    --streaming

```

On optimized hardware, expect:
- **Time-to-first-audio**: ~100 ms
- **Real-time factor (RTF)**: < 0.2
- **Throughput**: > 3,000 audio tokens/second

## Summary

- **SGLang streaming** reduces Fish-Speech inference latency by treating the Dual-AR transformer as a standard LLM and applying kernel-level optimizations.
- **Key techniques** include continuous batching, paged KV-cache, CUDA-graph replay, and Radix-Attention prefix caching.
- **Implementation** requires deploying the SGLang-Omni server, launching Fish-Speech with `--compile`, and enabling `streaming=True` in requests.
- **Performance targets** achieve ~100 ms time-to-first-audio and sub-0.2 RTF on modern GPUs by streaming raw PCM chunks via `InferenceResult` codes (`header`, `segment`, `final`).

## Frequently Asked Questions

### What is the expected time-to-first-audio with SGLang streaming?

With SGLang streaming enabled on an NVIDIA H200 GPU, Fish-Speech achieves approximately **100 milliseconds** to the first audio chunk. This is accomplished by bypassing full-file generation and immediately streaming PCM segments as they are decoded by the LLaMA-style transformer.

### How does the Dual-AR architecture benefit from LLM serving optimizations?

Fish-Speech uses a **dual-autoregressive** design where a semantic LLaMA decoder generates tokens that are then converted to audio. Because the token-generation loop is mathematically identical to standard large-language-model inference, it inherits optimizations like **continuous batching**, **paged KV-cache**, and **CUDA-graph replay** from SGLang without architectural changes.

### Can I use SGLang streaming with the WebUI or only via API?

SGLang streaming is primarily exposed through the **API layer** ([`tools/server/views.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/server/views.py)) and the command-line client ([`tools/api_client.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/api_client.py)). While the WebUI ([`tools/run_webui.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/run_webui.py)) supports the `--compile` flag for kernel fusion, real-time streaming playback requires consuming the chunked HTTP response programmatically or via the provided CLI client with the `--streaming` flag.

### What hardware requirements are needed for sub-200ms RTF?

Achieving a **real-time factor (RTF) below 0.2** and throughput exceeding **3,000 audio tokens per second** requires a modern NVIDIA GPU such as the **H200** or **H100**. The SGLang optimizations reduce CPU overhead and kernel launch latency, but saturating the continuous batching and paged attention mechanisms still demands high memory bandwidth and compute capacity typical of data-center GPUs.