How to Handle Multilingual TTS Without Phoneme Preprocessing in Fish-Speech

Fish-Speech synthesizes speech in dozens of languages directly from raw Unicode text using a unified tokenizer and end-to-end training, eliminating the need for grapheme-to-phoneme (G2P) conversion.

The fishaudio/fish-speech repository provides a text-to-speech system engineered for multilingual synthesis without traditional phoneme preprocessing. Unlike conventional TTS pipelines that require language-specific G2P front-ends, Fish-Speech processes raw Unicode characters through a unified tokenizer. This architecture allows the model to handle multilingual TTS without phoneme preprocessing by learning implicit character-to-sound mappings directly from training data.

Unified Tokenizer Architecture

UTF-8 Sub-word Processing

The core of Fish-Speech's multilingual capability resides in fish_speech/tokenizer.py. The system loads a single AutoTokenizer containing a fixed vocabulary of special tokens (e.g., <|endoftext|>, <|im_start|>, <|semantic:…|>) alongside language-agnostic word-piece tokens. Because the tokenizer operates on UTF-8 sub-words, characters from any supported script—Latin, Cyrillic, Chinese, Japanese, Korean, Arabic—are tokenized without language-specific rules.

Optional Phoneme Tokens

While the tokenizer defines optional phoneme markers (<|phoneme_start|> and <|phoneme_end|>), these are never required for standard inference. The model's architecture treats these as reserved tokens for potential future use, but the default multilingual TTS pipeline operates entirely on raw character tokens.

Language Code Validation and Mapping

The system maintains strict language code validation in fish_speech/utils/file.py. A helper function maps three-letter language codes (zh, jp, en) to the list of languages the model understands for each entry. During dataset loading via load_filelist, the code asserts that only supported language codes appear, ensuring the model receives only known language identifiers without consulting external G2P tables.

Text Cleaning Pipeline

Basic text sanitization occurs in fish_speech/text/clean.py. The pipeline removes emojis and normalizes a handful of punctuation symbols while preserving language-specific characters. This guarantees that input fed to the tokenizer remains safe yet retains the orthographic features necessary for the model to infer pronunciation. The cleaning process takes place before tokenization, ensuring raw Unicode strings are ready for the unified tokenizer.

Why Phoneme Preprocessing Is Unnecessary

Fish-Speech eliminates the need for phoneme preprocessing through three architectural decisions:

End-to-End Learning: The dual-autoregressive backbone (slow semantic + fast acoustic) is trained on raw text/audio pairs spanning over 10 million hours across approximately 50 languages. The model learns an implicit mapping from characters to acoustic patterns jointly, removing the need for an explicit phoneme stage.

Language-Agnostic Sampling: Sampling functions such as logits_to_probs and sample operate on the full vocabulary without language conditioning. The model's internal bias (semantic_logit_bias) restricts generation to the appropriate semantic token range regardless of the input language.

No Mandatory Phoneme Vocabulary: As confirmed in fish_speech/tokenizer.py, the tokenizer's special phoneme markers are defined but not used in the standard inference path. The system processes UTF-8 sub-words directly, allowing the attention layers to discover language-specific pronunciations internally.

Implementation Examples

Command-Line Inference

To generate speech in any supported language, pass the raw Unicode string directly to the CLI. The following examples demonstrate Chinese, Japanese, and Arabic synthesis without phoneme conversion:

# Chinese – 简体中文

python -m fish_speech.main \
  --text "<|speaker:0|>今天天气很好，我想去散步。" \
  --checkpoint-path checkpoints/s2-pro \
  --output output_zh.wav

# Japanese – 日本語

python -m fish_speech.main \
  --text "<|speaker:0|>今日はとても暑いです。" \
  --checkpoint-path checkpoints/s2-pro \
  --output output_ja.wav

# Arabic – العربية

python -m fish_speech.main \
  --text "<|speaker:0|>الجو جميل اليوم." \
  --checkpoint-path checkpoints/s2-pro \
  --output output_ar.wav

All three calls use the same binary; the tokenizer in fish_speech/tokenizer.py internally produces the correct sub-word IDs for each script.

Python API Usage

For programmatic access, instantiate the TTSInferenceEngine and pass raw Unicode text:

from fish_speech.inference_engine import TTSInferenceEngine
from pathlib import Path

engine = TTSInferenceEngine(
    checkpoint_path=Path("checkpoints/s2-pro"),
    device="cuda"
)

# Any Unicode string works – no phoneme conversion step.

audio = engine.generate(
    text="<|speaker:0|>Bonjour, comment ça va ?",
    max_new_tokens=400,
    temperature=0.9,
)

audio.save_wav("output_fr.wav")

Adding Custom Reference Voices

Even when adding reference voices for new languages, no phoneme preprocessing is required. The ReferenceLoader in fish_speech/inference_engine/reference_loader.py stores raw audio and its transcription:

from fish_speech.inference_engine.reference_loader import ReferenceLoader

loader = ReferenceLoader()
loader.add_reference(
    id="spanish_male",
    wav_file_path="samples/spanish_male.wav",
    reference_text="Este es un ejemplo de referencia en español."
)

The reference loader only stores the raw audio and its text; it never expects a phoneme representation.

Key Source Files

The following files implement the multilingual TTS pipeline without phoneme preprocessing:

File	Purpose
fish_speech/tokenizer.py	Defines the unified AutoTokenizer with UTF-8 sub-word vocabulary and optional phoneme tokens.
fish_speech/utils/file.py	Contains the language code validation and load_filelist function for dataset loading.
fish_speech/text/clean.py	Implements basic text sanitization (emoji removal, punctuation normalization) while preserving language characters.
fish_speech/i18n/core.py	Handles UI localization strings, separate from the TTS synthesis pipeline.
fish_speech/inference_engine/reference_loader.py	Manages reference voice loading without requiring phoneme input.
README.md	Documents the multilingual support and training data coverage (>10M hours, ~50 languages).

Summary

• Fish-Speech processes multilingual TTS without phoneme preprocessing by using a unified tokenizer that operates directly on UTF-8 sub-words.
• The system validates language codes in fish_speech/utils/file.py but never requires external grapheme-to-phoneme (G2P) tables.
• End-to-end training on raw text/audio pairs allows the model to learn implicit character-to-sound mappings across approximately 50 languages.
• Optional phoneme tokens exist in the vocabulary but are not used in standard inference, as confirmed in fish_speech/tokenizer.py.
• You can synthesize speech in Chinese, Japanese, Arabic, Spanish, or any supported language by passing raw Unicode strings directly to the CLI or Python API.

Frequently Asked Questions

Does Fish-Speech require phoneme dictionaries for new languages?

No. Fish-Speech does not use phoneme dictionaries or G2P front-ends. The model learns pronunciation directly from character sequences during training. You can add new languages by training on raw text/audio pairs; the unified tokenizer in fish_speech/tokenizer.py will handle the new Unicode characters automatically.

What happens if I include phoneme markers in the input text?

The tokenizer recognizes optional phoneme tokens (<|phoneme_start|> and <|phoneme_end|>), but these are reserved for potential future use. Including them in standard inference will not trigger phoneme processing; the model treats them as regular tokens. The standard pipeline ignores these markers and processes the raw text directly.

How does the model handle languages with non-Latin scripts like Arabic or Chinese?

The unified tokenizer processes UTF-8 sub-words, which means characters from Arabic, Chinese, Japanese, Korean, Cyrillic, and other scripts are tokenized into the same semantic space as Latin characters. The text cleaning module in fish_speech/text/clean.py removes emojis and normalizes punctuation while preserving script-specific characters, allowing the model to generate appropriate acoustic patterns for each writing system.

Can I fine-tune Fish-Speech on a specific language without modifying the tokenizer?

Yes. Because the tokenizer vocabulary is fixed and language-agnostic, you can fine-tune on new languages or domains by preparing a dataset with the target language's raw text. The load_filelist function in fish_speech/utils/file.py validates language codes but does not require phoneme annotations. Simply ensure your text files contain valid Unicode for the target language and reference the appropriate language code in your file list.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}