# How to Handle Long-Context Multi-Turn TTS Generation in Fish-Speech

> Learn to handle long-context multi-turn TTS generation with Fish-Speech. Our stateful pipeline processes dialogue batches, maintaining context across turns for seamless audio.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Fish-Speech handles long-context multi-turn TTS generation by splitting speaker-tagged dialogue into byte-limited batches and processing them through a stateful conversation pipeline that maintains context across turns.**

The fishaudio/fish-speech repository implements a conversation-driven inference pipeline that treats every TTS request as a sequence of messages. When input text exceeds the model's context window, the library automatically segments the dialogue into speaker-tagged turns, groups them into manageable batches, and feeds each batch to the model while preserving multi-turn coherence.

## Core Architecture for Long-Context Generation

### Speaker-Aware Text Segmentation

The pipeline begins in [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) where the `split_text_by_speaker()` function (lines 554-574) scans raw input strings for tags of the form `<|speaker:X|>`. This function returns a list of individual turns, ensuring that speaker boundaries are respected and preventing mid-sentence truncation.

### Byte-Limited Batching

Immediately following segmentation, `group_turns_into_batches()` (lines 585-605 in the same file) aggregates consecutive turns until either the maximum number of speakers (`max_speakers`, default 3) or the maximum byte size (`max_bytes`, default 300) is reached. The CLI flag `--chunk-length` directly overrides the `max_bytes` parameter, giving users control over batch granularity.

### Stateful Conversation Management

The `Conversation` class defined in [`fish_speech/conversation.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/conversation.py) (lines 19-80) maintains the dialogue state across batches. It assembles lists of `Message` objects, inserts special *IM_START* and *IM_END* tokens, and produces a `ContentSequence` that encodes itself for model inference. This design ensures that previously generated speech (stored as assistant VQ codes) remains visible in the context window for subsequent batches.

## The generate_long() Generator Workflow

The main orchestration happens in `generate_long()` (lines 620-740 of [`inference.py`](https://github.com/fishaudio/fish-speech/blob/main/inference.py)). This function is a **generator** that yields `GenerateResponse` objects, enabling both streaming to a UI and batch-mode execution.

The workflow follows these steps:

1. **Parse and batch** – Calls `split_text_by_speaker()` and `group_turns_into_batches()` to create manageable input chunks.
2. **Initialize base conversation** – Creates a system message containing the conversion prompt once, reusing it for every batch.
3. **Iterate over batches** – For each batch it:
   - Appends a **user** message with the batch text.
   - Creates an **assistant** placeholder (role = "assistant", modality = "voice").
   - Calls `conversation_gen.encode_for_inference()` to obtain the tokenised prompt.
   - Runs the **decoder** (`generate()`) to obtain VQ-code tensors for the spoken output.
   - Appends a **assistant** message containing the generated VQ codes so the next batch sees the previous speech as part of the context.
4. **Yield results** – After each batch it yields a `GenerateResponse(action="sample", codes=..., text=...)`. When all batches are processed it yields a final `GenerateResponse(action="next")`.

## Configuration Parameters for Context Management

| Parameter | Effect |
|-----------|--------|
| `--chunk-length` (or `chunk_length` arg) | Maximum UTF-8 bytes per batch – larger values mean fewer round-trips but higher risk of hitting the model's max-seq limit. |
| `--max-new-tokens` | Upper bound on tokens the decoder may emit per batch. Set to 0 for "no limit". |
| `--top-p`, `--top-k`, `--temperature` | Sampling controls for audio-generation quality. |
| `--iterative-prompt` | When `False`, each batch starts from the original system prompt only (no history). Useful for memory-constrained environments. |
| `--num-samples` | Number of independent samples to generate for the *entire* input (each sample runs through all batches). |

## Practical Implementation Examples

### CLI Usage

Generate long-form speech directly from the command line:

```bash
python -m fish_speech.models.text2semantic.inference \
    --text "<|speaker:0|>Hello! How are you?<|speaker:1|>I am fine, thanks. Let me tell you a story that is a bit longer than usual..." \
    --chunk-length 400 \
    --max-new-tokens 1024 \
    --output-dir ./tts-output

```

The CLI automatically splits the input, streams batch-wise results, and writes the final waveform to `./tts-output`.

### Python API Usage

For programmatic control, use the `generate_long` generator:

```python
import torch
from fish_speech.models.text2semantic.inference import (
    generate_long,
    split_text_by_speaker,
    group_turns_into_batches,
)
from fish_speech.utils.logging_utils import logger
from fish_speech.models.text2semantic.inference import init_model

# 1️⃣ Load model once

model, decode_one_token = init_model(
    checkpoint_path="checkpoints/s2-pro",
    device="cuda",
    precision=torch.bfloat16,
    compile=False,
)

# 2️⃣ Prepare long text with speaker tags

long_text = """
<|speaker:0|>Hi, I'm Alice.
<|speaker:1|>Hello Alice, I'm Bob. Let's discuss the project…
... (very long conversation) ...
"""

# 3️⃣ (optional) manually inspect how the library would split it

turns = split_text_by_speaker(long_text)
batches = group_turns_into_batches(turns, max_speakers=3, max_bytes=300)
print("Will be processed in", len(batches), "batches")

# 4️⃣ Run the generator

for chunk in generate_long(
    model=model,
    device="cuda",
    decode_one_token=decode_one_token,
    text=long_text,
    max_new_tokens=1024,
    top_p=0.9,
    top_k=30,
    temperature=1.0,
    chunk_length=300,          # same as max_bytes above

    iterative_prompt=True,
):
    if chunk.action == "sample":
        # `chunk.codes` is a tensor (num_codebooks, T) of VQ indices for this batch

        logger.info(f"Generated {chunk.codes.shape[-1]} tokens for batch")
        # Example: decode to audio (requires the VQ decoder)

        # wav = decode_vq_codes(chunk.codes)

    else:
        logger.info("All batches processed")

```

### Manual Batch Inspection

Inspect how text is segmented before generation:

```python
from fish_speech.models.text2semantic.inference import split_text_by_speaker, group_turns_into_batches

raw = "<|speaker:0|>First turn.<|speaker:1|>Second turn that is quite long..."
turns = split_text_by_speaker(raw)

# ['<|speaker:0|>First turn.', '<|speaker:1|>Second turn that is quite long...']

batches = group_turns_into_batches(turns, max_speakers=2, max_bytes=100)

# ['<|speaker:0|>First turn.\n<|speaker:1|>Second turn that is quite long...']

```

## Key Source Files

| File | Role | Direct link |
|------|------|-------------|
| [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) | Implements `generate_long`, the long-context orchestration, and the helper functions `split_text_by_speaker` & `group_turns_into_batches`. | [view on GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) |
| [`fish_speech/conversation.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/conversation.py) | Defines `Conversation`, `Message`, and the encoding/visualisation logic that `generate_long` relies on. | [view on GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/conversation.py) |
| [`fish_speech/content_sequence.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/content_sequence.py) | Low-level representation of a sequence of text/audio parts; handles tokenisation, loss-mask creation, and VQ-part handling. | [view on GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/content_sequence.py) |
| [`tools/api_server.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/api_server.py) & [`tools/api_client.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/api_client.py) | Provide a HTTP wrapper around the same generation pipeline – useful for remote services. | [server](https://github.com/fishaudio/fish-speech/blob/main/tools/api_server.py) • [client](https://github.com/fishaudio/fish-speech/blob/main/tools/api_client.py) |
| [`tools/webui/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py) | Example web UI that streams the generator output to a browser. | [view on GitHub](https://github.com/fishaudio/fish-speech/blob/main/tools/webui/inference.py) |

## Summary

- **Fish-Speech** treats long-context multi-turn TTS as a conversation pipeline, where each batch of text is processed as a user message and generated audio is appended back as an assistant message.
- **Speaker-aware segmentation** via `split_text_by_speaker()` ensures dialogue structure is preserved when splitting long inputs.
- **Byte-limited batching** through `group_turns_into_batches()` guarantees that the tokenized prompt never exceeds the model's `max_seq_len`, preventing out-of-memory errors.
- **Stateful context** is maintained by appending generated VQ codes back to the `Conversation` object, allowing subsequent batches to reference previous speech for prosodic consistency.
- **Streaming interface** – the `generate_long()` generator yields `GenerateResponse` objects after each batch, enabling real-time audio streaming in web UIs.

## Frequently Asked Questions

### How does Fish-Speech prevent context window overflow during long-form generation?

Fish-Speech prevents overflow by implementing **byte-limited batching** in `group_turns_into_batches()` (lines 585-605 of [`inference.py`](https://github.com/fishaudio/fish-speech/blob/main/inference.py)). This function aggregates consecutive speaker turns until either the `max_speakers` limit (default 3) or the `max_bytes` ceiling (default 300) is reached. Because the tokenizer maps roughly one byte to one token, this guarantees that the final prompt (system message + history + current batch) fits within the model's `max_seq_len`.

### What are speaker tags and why are they required for multi-turn TTS?

Speaker tags are special tokens of the form `<|speaker:X|>` that demarcate turn boundaries in the input text. They are required because Fish-Speech's `split_text_by_speaker()` function (lines 554-574) uses these tags to segment the input into discrete turns before batching. Without explicit tags, the system cannot distinguish between different speakers or determine safe truncation points, which risks cutting off mid-sentence and losing prosodic context between dialogue participants.

### Can I disable iterative prompting to reduce memory consumption?

Yes. The `--iterative-prompt` flag (default `True`) controls whether the conversation history is carried forward between batches. When set to `False`, each batch starts fresh from the original system prompt only, effectively disabling the stateful context mechanism. This reduces GPU memory usage significantly for very long inputs, though it may degrade prosodic continuity across batch boundaries since the model loses access to previously generated speech.

### How do I tune the batch size for my specific hardware?

Batch granularity is controlled by the `--chunk-length` argument (Python parameter `chunk_length`), which maps directly to the `max_bytes` parameter in `group_turns_into_batches()`. For high-memory GPUs (e.g., A100), increase `--chunk-length` to 800-1000 bytes to reduce the number of forward passes and improve throughput. For consumer GPUs with limited VRAM, reduce it to 200-300 bytes to prevent out-of-memory errors during attention computation. Monitor the `max_new_tokens` parameter as well, as it caps the decoder output per batch independently of the input chunk size.