How to Handle Long-Context Multi-Turn TTS Generation in Fish-Speech

Fish-Speech handles long-context multi-turn TTS generation by splitting speaker-tagged dialogue into byte-limited batches and processing them through a stateful conversation pipeline that maintains context across turns.

The fishaudio/fish-speech repository implements a conversation-driven inference pipeline that treats every TTS request as a sequence of messages. When input text exceeds the model's context window, the library automatically segments the dialogue into speaker-tagged turns, groups them into manageable batches, and feeds each batch to the model while preserving multi-turn coherence.

Core Architecture for Long-Context Generation

Speaker-Aware Text Segmentation

The pipeline begins in fish_speech/models/text2semantic/inference.py where the split_text_by_speaker() function (lines 554-574) scans raw input strings for tags of the form <|speaker:X|>. This function returns a list of individual turns, ensuring that speaker boundaries are respected and preventing mid-sentence truncation.

Byte-Limited Batching

Immediately following segmentation, group_turns_into_batches() (lines 585-605 in the same file) aggregates consecutive turns until either the maximum number of speakers (max_speakers, default 3) or the maximum byte size (max_bytes, default 300) is reached. The CLI flag --chunk-length directly overrides the max_bytes parameter, giving users control over batch granularity.

Stateful Conversation Management

The Conversation class defined in fish_speech/conversation.py (lines 19-80) maintains the dialogue state across batches. It assembles lists of Message objects, inserts special IM_START and IM_END tokens, and produces a ContentSequence that encodes itself for model inference. This design ensures that previously generated speech (stored as assistant VQ codes) remains visible in the context window for subsequent batches.

The generate_long() Generator Workflow

The main orchestration happens in generate_long() (lines 620-740 of inference.py). This function is a generator that yields GenerateResponse objects, enabling both streaming to a UI and batch-mode execution.

The workflow follows these steps:

1. Parse and batch – Calls split_text_by_speaker() and group_turns_into_batches() to create manageable input chunks.
2. Initialize base conversation – Creates a system message containing the conversion prompt once, reusing it for every batch.
3. Iterate over batches – For each batch it:
   • Appends a user message with the batch text.
   • Creates an assistant placeholder (role = "assistant", modality = "voice").
   • Calls conversation_gen.encode_for_inference() to obtain the tokenised prompt.
   • Runs the decoder (generate()) to obtain VQ-code tensors for the spoken output.
   • Appends a assistant message containing the generated VQ codes so the next batch sees the previous speech as part of the context.
4. Yield results – After each batch it yields a GenerateResponse(action="sample", codes=..., text=...). When all batches are processed it yields a final GenerateResponse(action="next").

Configuration Parameters for Context Management

Parameter	Effect
--chunk-length (or chunk_length arg)	Maximum UTF-8 bytes per batch – larger values mean fewer round-trips but higher risk of hitting the model's max-seq limit.
--max-new-tokens	Upper bound on tokens the decoder may emit per batch. Set to 0 for "no limit".
--top-p, --top-k, --temperature	Sampling controls for audio-generation quality.
--iterative-prompt	When False, each batch starts from the original system prompt only (no history). Useful for memory-constrained environments.
--num-samples	Number of independent samples to generate for the entire input (each sample runs through all batches).

Practical Implementation Examples

CLI Usage

Generate long-form speech directly from the command line:

python -m fish_speech.models.text2semantic.inference \
    --text "<|speaker:0|>Hello! How are you?<|speaker:1|>I am fine, thanks. Let me tell you a story that is a bit longer than usual..." \
    --chunk-length 400 \
    --max-new-tokens 1024 \
    --output-dir ./tts-output

The CLI automatically splits the input, streams batch-wise results, and writes the final waveform to ./tts-output.

Python API Usage

For programmatic control, use the generate_long generator:

import torch
from fish_speech.models.text2semantic.inference import (
    generate_long,
    split_text_by_speaker,
    group_turns_into_batches,
)
from fish_speech.utils.logging_utils import logger
from fish_speech.models.text2semantic.inference import init_model

# 1️⃣ Load model once

model, decode_one_token = init_model(
    checkpoint_path="checkpoints/s2-pro",
    device="cuda",
    precision=torch.bfloat16,
    compile=False,
)

# 2️⃣ Prepare long text with speaker tags

long_text = """
<|speaker:0|>Hi, I'm Alice.
<|speaker:1|>Hello Alice, I'm Bob. Let's discuss the project…
... (very long conversation) ...
"""

# 3️⃣ (optional) manually inspect how the library would split it

turns = split_text_by_speaker(long_text)
batches = group_turns_into_batches(turns, max_speakers=3, max_bytes=300)
print("Will be processed in", len(batches), "batches")

# 4️⃣ Run the generator

for chunk in generate_long(
    model=model,
    device="cuda",
    decode_one_token=decode_one_token,
    text=long_text,
    max_new_tokens=1024,
    top_p=0.9,
    top_k=30,
    temperature=1.0,
    chunk_length=300,          # same as max_bytes above

    iterative_prompt=True,
):
    if chunk.action == "sample":
        # `chunk.codes` is a tensor (num_codebooks, T) of VQ indices for this batch

        logger.info(f"Generated {chunk.codes.shape[-1]} tokens for batch")
        # Example: decode to audio (requires the VQ decoder)

        # wav = decode_vq_codes(chunk.codes)

    else:
        logger.info("All batches processed")

Manual Batch Inspection

Inspect how text is segmented before generation:

from fish_speech.models.text2semantic.inference import split_text_by_speaker, group_turns_into_batches

raw = "<|speaker:0|>First turn.<|speaker:1|>Second turn that is quite long..."
turns = split_text_by_speaker(raw)

# ['<|speaker:0|>First turn.', '<|speaker:1|>Second turn that is quite long...']

batches = group_turns_into_batches(turns, max_speakers=2, max_bytes=100)

# ['<|speaker:0|>First turn.\n<|speaker:1|>Second turn that is quite long...']

Key Source Files

File	Role	Direct link
fish_speech/models/text2semantic/inference.py	Implements generate_long, the long-context orchestration, and the helper functions split_text_by_speaker & group_turns_into_batches.	view on GitHub
fish_speech/conversation.py	Defines Conversation, Message, and the encoding/visualisation logic that generate_long relies on.	view on GitHub
fish_speech/content_sequence.py	Low-level representation of a sequence of text/audio parts; handles tokenisation, loss-mask creation, and VQ-part handling.	view on GitHub
tools/api_server.py & tools/api_client.py	Provide a HTTP wrapper around the same generation pipeline – useful for remote services.	server • client
tools/webui/inference.py	Example web UI that streams the generator output to a browser.	view on GitHub

Summary

• Fish-Speech treats long-context multi-turn TTS as a conversation pipeline, where each batch of text is processed as a user message and generated audio is appended back as an assistant message.
• Speaker-aware segmentation via split_text_by_speaker() ensures dialogue structure is preserved when splitting long inputs.
• Byte-limited batching through group_turns_into_batches() guarantees that the tokenized prompt never exceeds the model's max_seq_len, preventing out-of-memory errors.
• Stateful context is maintained by appending generated VQ codes back to the Conversation object, allowing subsequent batches to reference previous speech for prosodic consistency.
• Streaming interface – the generate_long() generator yields GenerateResponse objects after each batch, enabling real-time audio streaming in web UIs.

Frequently Asked Questions

How does Fish-Speech prevent context window overflow during long-form generation?

Fish-Speech prevents overflow by implementing byte-limited batching in group_turns_into_batches() (lines 585-605 of inference.py). This function aggregates consecutive speaker turns until either the max_speakers limit (default 3) or the max_bytes ceiling (default 300) is reached. Because the tokenizer maps roughly one byte to one token, this guarantees that the final prompt (system message + history + current batch) fits within the model's max_seq_len.

What are speaker tags and why are they required for multi-turn TTS?

Speaker tags are special tokens of the form <|speaker:X|> that demarcate turn boundaries in the input text. They are required because Fish-Speech's split_text_by_speaker() function (lines 554-574) uses these tags to segment the input into discrete turns before batching. Without explicit tags, the system cannot distinguish between different speakers or determine safe truncation points, which risks cutting off mid-sentence and losing prosodic context between dialogue participants.

Can I disable iterative prompting to reduce memory consumption?

Yes. The --iterative-prompt flag (default True) controls whether the conversation history is carried forward between batches. When set to False, each batch starts fresh from the original system prompt only, effectively disabling the stateful context mechanism. This reduces GPU memory usage significantly for very long inputs, though it may degrade prosodic continuity across batch boundaries since the model loses access to previously generated speech.

How do I tune the batch size for my specific hardware?

Batch granularity is controlled by the --chunk-length argument (Python parameter chunk_length), which maps directly to the max_bytes parameter in group_turns_into_batches(). For high-memory GPUs (e.g., A100), increase --chunk-length to 800-1000 bytes to reduce the number of forward passes and improve throughput. For consumer GPUs with limited VRAM, reduce it to 200-300 bytes to prevent out-of-memory errors during attention computation. Monitor the max_new_tokens parameter as well, as it caps the decoder output per batch independently of the input chunk size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}