How to Implement Multi-Speaker Voice Generation with Speaker Tokens in Fish-Speech

Fish-Speech implements multi-speaker voice generation by reserving special inline tokens of the form <|speaker:ID|> that are inserted into the token stream via the ContentSequence class, allowing the transformer model to condition acoustic generation on speaker context without requiring a separate speaker embedding table.

Fish-Speech, the open-source text-to-speech system developed by fishaudio/fish-speech, enables multi-speaker voice generation through a unique token-based conditioning mechanism. Unlike conventional TTS architectures that rely on external speaker embedding networks, Fish-Speech treats speaker identities as discrete vocabulary items. This design allows the model to handle unlimited speaker switches within a single conversation by parsing special tokens directly in the input sequence.

Speaker Token Architecture

Fish-Speech represents speakers as special inline tokens formatted as <|speaker:ID|>, where the ID can be an integer, string identifier (e.g., "user", "assistant"), or any custom label. These tokens are processed as ordinary vocabulary items by the tokenizer, ensuring seamless integration with the language model's attention mechanism.

Token Definitions and Parsing

In fish_speech/tokenizer.py, the FishTokenizer class handles speaker tokens alongside other special modality tokens. While MODALITY_TOKENS defines stream types like <|text|> and <|voice|>, speaker tokens are constructed dynamically based on the speaker identifier.

# fish_speech/tokenizer.py

MODALITY_TOKENS = {
    "text": "<|text|>",
    "voice": "<|voice|>",
    "interleave": "<|interleave|>",
}

# Speaker tokens are built on-the-fly, e.g. "<|speaker:0|>", "<|speaker:user|>"

The tokenizer processes these tokens because FishTokenizer.encode() is configured with allowed_special="all", converting strings like <|speaker:alice|> into unique token IDs that the model attends to during generation.

Building Sequences with ContentSequence

The ContentSequence class in fish_speech/content_sequence.py serves as the primary builder for multi-speaker prompts. Its append() method accepts an optional speaker argument that automatically injects the corresponding token before the content.

# fish_speech/content_sequence.py

def append(self, part_or_parts, add_end=False, speaker=None):
    parts_to_add = [part_or_parts] if not isinstance(part_or_parts, list) else part_or_parts
    if speaker is not None:
        speaker_token = f"<|speaker:{speaker}|>"
        self.parts.append(TextPart(text=speaker_token))
    self.parts.extend(parts_to_add)
    ...

This produces interleaved sequences following the pattern:

<|interleave|><|speaker:1|> TEXT AUDIO ề<|speaker:2|> TEXT AUDIO ề

The <|interleave|> token signals the model to expect alternating text and audio content, while each <|speaker:X|> token establishes voice context for subsequent tokens until the next speaker marker appears.

Training Data Preparation

During dataset construction in fish_speech/datasets/semantic.py, the pack_sentences function explicitly inserts speaker tokens to create multi-turn training examples. The use_speaker parameter controls this behavior, supporting both deterministic insertion and random dropout for robustness training.

# fish_speech/datasets/semantic.py (pack_sentences)

seq.append(TextPart(text=f"<|speaker:user|> {cated_sentences}"), add_end=True)

# assistant turn – voice token + speaker token

seq.append(
    [TextPart(text="<|speaker:assistant|> <|voice|>"), vq_part],
    add_end=True,
)

When use_speaker=True (the default), every utterance receives a speaker label. Setting use_speaker to a float value (e.g., 0.3) randomly drops speaker tokens with that probability, teaching the model to generate consistent voices even when labels are missing.

Inference-Time Speaker Handling

For generation, Fish-Speech provides utilities in fish_speech/models/text2semantic/inference.py to parse long prompts containing multiple speakers and structure them for efficient batching.

Splitting Conversations by Speaker

The split_text_by_speaker() function uses regular expressions to segment input text at speaker token boundaries, isolating individual turns for processing.

# fish_speech/models/text2semantic/inference.py

def split_text_by_speaker(text: str) -> list[str]:
    pattern = r"(<\|speaker:\d+\|>)"
    parts = re.split(pattern, text)
    # …assemble <|speaker|> + following text into a turn…

This ensures that speaker context is preserved when feeding long conversations to the model incrementally.

Batching Multi-Speaker Turns

The group_turns_into_batches() function organizes speaker turns into batches respecting constraints on speaker count and sequence length.

# fish_speech/models/text2semantic/inference.py

def group_turns_into_batches(turns, max_speakers=3, max_bytes=300):
    # …batch logic that keeps speaker changes together…

These utilities power the high-level generate_long() routine, enabling the model to process extended multi-speaker dialogues while maintaining voice consistency within each speaker's segments.

Complete Implementation Example

The following end-to-end example demonstrates building a three-speaker conversation, encoding it with speaker tokens, and passing it to the Fish-Speech generation pipeline.

# Example: multi-speaker generation

from fish_speech.content_sequence import ContentSequence, TextPart
from fish_speech.tokenizer import FishTokenizer
from fish_speech.models.text2semantic.inference import generate_long

# 1️⃣ Load tokenizer & model (model loading omitted for brevity)

tokenizer = FishTokenizer("path/to/model")
model = ...   # a Text2Semantic model exposing .tokenizer and .config

# 2️⃣ Build the prompt

seq = ContentSequence(modality="interleave")
seq.append(TextPart(text="Hello, I am Alice."), speaker="alice")
seq.append(TextPart(text="Hi Alice, I am Bob."), speaker="bob")
seq.append(
    [TextPart(text="<|voice|>"), TextPart(text="Here's my reply.")],
    speaker="bob",
    add_end=True,
)

# 3️⃣ Encode to token tensors

encoded = seq.encode(tokenizer)

# 4️⃣ Run generation (single-turn demo)

outputs = generate_long(
    model=model,
    device="cpu",
    decode_one_token=model.decode_one_token,   # model-specific helper

    text=seq.encode(tokenizer).tokens.tolist(),
    max_new_tokens=256,
    top_p=0.9,
    temperature=1.0,
)

print("Generated token IDs:", outputs.codes)

The speaker tokens (<|speaker:alice|>, <|speaker:bob|>) are automatically converted to token IDs by FishTokenizer.encode() and guide the model's voice selection throughout the sequence.

Practical Code Snippets

Adding Speaker Tokens Manually

For custom prompt construction, manually append speaker tokens using ContentSequence:

from fish_speech.content_sequence import ContentSequence, TextPart

seq = ContentSequence(modality="interleave")
seq.append(TextPart(text="Good morning!"), speaker="john")
seq.append(TextPart(text="Morning, John."), speaker="mary")
seq.append(TextPart(text="Let's start."), speaker="john", add_end=True)

Configuring the Dataset for Speaker Labels

Enable automatic speaker token insertion when loading training data:

from fish_speech.datasets.semantic import AutoTextSemanticInstructionDataset
from fish_speech.tokenizer import FishTokenizer

tokenizer = FishTokenizer("model_path")
dataset = AutoTextSemanticInstructionDataset(
    proto_files=["data/*.proto"],
    tokenizer=tokenizer,
    use_speaker=True,          # embeds <|speaker:…|> tokens

    interactive_prob=1.0,      # forces interleaved mode

)

sample = dataset[0]           # returns {"tokens": ..., "labels": ...}

Batch Generation with Speaker Awareness

Process multi-speaker prompts efficiently using inference utilities:

from fish_speech.models.text2semantic.inference import split_text_by_speaker, group_turns_into_batches

prompt = "<|speaker:alice|> How are you?<|speaker:bob|> Fine, thanks!"
turns = split_text_by_speaker(prompt)
batches = group_turns_into_batches(turns, max_speakers=2, max_bytes=500)

for batch in batches:
    # feed each batch to `generate_long` …

    pass

Summary

• Fish-Speech uses inline speaker tokens (<|speaker:ID|>) to identify voices, eliminating the need for external speaker embedding tables.
• The ContentSequence class in fish_speech/content_sequence.py automates token insertion via the speaker parameter in append().
• Training data preparation in fish_speech/datasets/semantic.py supports configurable speaker labels with the use_speaker flag, including random dropout for robustness.
• Inference utilities split_text_by_speaker() and group_turns_into_batches() in fish_speech/models/text2semantic/inference.py handle complex multi-turn dialogues.
• The tokenizer treats speaker tokens as ordinary special tokens when allowed_special="all" is set, enabling seamless integration with the transformer architecture.

Frequently Asked Questions

How are speaker tokens formatted in Fish-Speech?

Speaker tokens follow the format <|speaker:ID|> where ID can be any string or integer identifier, such as <|speaker:0|>, <|speaker:user|>, or <|speaker:alice|>. These are constructed dynamically and parsed by FishTokenizer.encode() as standard special tokens.

Can I use custom speaker IDs beyond integers?

Yes. While numeric IDs are common, Fish-Speech supports arbitrary string identifiers like "assistant" or "user". The ContentSequence.append() method accepts any string for the speaker argument and formats it into the token structure automatically.

How does the model learn distinct voices without a separate speaker embedding table?

Fish-Speech treats speaker tokens as unique vocabulary items within the standard language model embedding table. Because each speaker token receives a distinct token ID, the transformer attention layers learn to associate specific acoustic patterns with these discrete identifiers, allowing voice conditioning through the existing token embedding mechanism.

What is the maximum number of speakers supported in a single generation?

There is no hardcoded limit enforced by the architecture. However, the group_turns_into_batches() utility accepts a max_speakers parameter (defaulting to 3) to manage memory and computational efficiency during inference. You can increase this value for dialogues with more participants, provided your hardware accommodates the resulting sequence length.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}