# How to Implement Custom Text Preprocessing Pipelines in Fish-Speech

> Learn how to implement custom text preprocessing pipelines in Fish-Speech. Easily modify existing functions or create your own to enhance your audio processing.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**You can implement custom text preprocessing in Fish-Speech by either modifying the `clean_text` function in [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py), creating a wrapper module that extends the default behavior, or subclassing the dataset classes to inject your own pipeline.**

Fish-Speech processes all input text through a centralized cleaning routine before tokenization and model inference. The `fishaudio/fish-speech` repository hard-codes the import of `clean_text` across dataset classes, which means implementing custom preprocessing requires strategic intervention in one of three specific locations.

## Understanding the Default Text Cleaning Pipeline

The default preprocessing logic lives in [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py). The `clean_text` function performs four core operations:

1. **Whitespace stripping** – Removes leading and trailing spaces.
2. **Punctuation normalization** – Replaces Chinese punctuation (e.g., `‘`, `’`) with ASCII equivalents.
3. **Emoji removal** – Strips Unicode emoji ranges using a compiled regex.
4. **Character deduplication** – Collapses repeated commas or periods into single instances.

Dataset classes such as `AutoTextSemanticInstructionDataset` and `AutoTextSemanticInstructionIterableDataset` import this function directly:

```python
from fish_speech.text.clean import clean_text

# Inside the dataset class

text = clean_text(random.choice(sentence.texts))

```

Because the import is hard-coded, the library does not expose a configuration hook for alternative pipelines.

## Three Methods to Implement Custom Text Preprocessing Pipelines

You have three idiomatic approaches to inject custom logic without breaking the core training loop.

### Method 1: Patch clean_text In-Place

**Best for:** Simple tweaks like adding punctuation mappings, extra regex filters, or whitespace normalization.

Edit [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py) directly to modify the `clean_text` function or the `SYMBOLS_MAPPING` dictionary:

```python

# fish_speech/text/clean.py

import re

# Extend the existing mapping

SYMBOLS_MAPPING = {
    "‘": "'",
    "’": "'",
    "，": ",",      # Full-width comma

    "。": ".",      # Full-width period

    "！": "!",      # Full-width exclamation

    "？": "?",      # Full-width question mark

}

REPLACE_SYMBOL_REGEX = re.compile("|".join(map(re.escape, SYMBOLS_MAPPING)))

def clean_text(text: str) -> str:
    """Standard Fish-Speech cleaning with custom additions."""
    text = text.strip()
    text = REPLACE_SYMBOL_REGEX.sub(lambda m: SYMBOLS_MAPPING[m.group()], text)
    
    # Remove emojis (existing logic)

    EMOJI_REGEX = re.compile("[\U0001f600-\U0001f64f...", flags=re.UNICODE)
    text = EMOJI_REGEX.sub("", text)
    
    # Collapse repeated punctuation

    text = re.sub(r"([,.])\1+", r"\1", text)
    
    # Custom: Normalize internal whitespace

    text = re.sub(r"\s+", " ", text)
    
    return text

```

All downstream datasets automatically inherit these changes.

### Method 2: Create a Wrapper Function

**Best for:** Domain-specific experiments that require preprocessing steps you do not want to commit to the core library.

Create a new module that imports the base cleaner and extends it:

```python

# fish_speech/text/custom_clean.py

import re
from .clean import clean_text as base_clean

def custom_clean(text: str) -> str:
    """Base cleaning + domain-specific processing."""
    # Run library defaults first

    text = base_clean(text)
    
    # Example: Remove all numeric digits for letter-only TTS

    text = re.sub(r"\d+", "", text)
    
    # Example: Convert to lowercase for case-insensitive models

    text = text.lower()
    
    # Example: Expand common abbreviations

    text = text.replace("Dr.", "Doctor")
    text = text.replace("Mr.", "Mister")
    
    return text

```

Then redirect the dataset imports to use your wrapper. Modify the import statement in [`fish_speech/datasets/semantic.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/datasets/semantic.py):

```diff
- from fish_speech.text.clean import clean_text
+ from fish_speech.text.custom_clean import custom_clean as clean_text

```

This approach preserves the original [`clean.py`](https://github.com/fishaudio/fish-speech/blob/main/clean.py) while allowing you to swap pipelines via import aliasing.

### Method 3: Subclass the Dataset

**Best for:** Completely separate preprocessing pipelines (e.g., language-specific tokenization) that must coexist with the default dataset.

Subclass `AutoTextSemanticInstructionDataset` and override the method responsible for text augmentation:

```python

# my_project/custom_dataset.py

import random
import torch
from fish_speech.datasets.semantic import AutoTextSemanticInstructionDataset
from fish_speech.text.custom_clean import custom_clean

class MySemanticDataset(AutoTextSemanticInstructionDataset):
    """
    Dataset that uses a custom preprocessing pipeline
    instead of the default clean_text.
    """
    
    def augment(self):
        """
        Override augment to inject custom_clean.
        """
        response = self.sample_data()
        if not response.samples:
            return None

        all_tokens, all_labels = [], []
        while response.samples:
            sentence = response.samples.pop(0)
            
            # Use custom preprocessing instead of clean_text

            text = custom_clean(random.choice(sentence.texts))

            tokens, labels = self.pack_sentences(
                sentences=[text],
                semantics=[sentence.semantics],
                skip_text=random.random() < self.skip_text_prob,
            )
            all_tokens.append(tokens)
            all_labels.append(labels)

        return {
            "tokens": torch.cat(all_tokens, dim=1),
            "labels": torch.cat(all_labels, dim=1)
        }

```

Instantiate your custom dataset in the training script:

```python
from my_project.custom_dataset import MySemanticDataset
from fish_speech.tokenizer import FishTokenizer

train_ds = MySemanticDataset(
    proto_files=["data/protos"],
    tokenizer=FishTokenizer("checkpoints/fish-speech-1.5/tokenizer.tiktoken"),
    interactive_prob=1.0,
)

```

This method isolates your changes to a specific experiment without affecting other dataset users.

## Key Source Files for Text Preprocessing

| File | Purpose | Location |
|------|---------|----------|
| [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py) | Core `clean_text` implementation with regex mappings and emoji removal | [GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py) |
| [`fish_speech/text/__init__.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/__init__.py) | Re-exports `clean_text`; modify imports here to redirect to custom wrappers | [GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/__init__.py) |
| [`fish_speech/datasets/semantic.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/datasets/semantic.py) | Dataset classes that consume `clean_text`; target for import overrides or subclassing | [GitHub](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/datasets/semantic.py) |

## Summary

- Fish-Speech centralizes text preprocessing in the `clean_text` function inside [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py), which applies whitespace stripping, punctuation normalization, emoji removal, and character deduplication.
- Because dataset classes import `clean_text` directly, you must use one of three strategies to customize: **in-place patching** for simple edits, **wrapper functions** for experimental pipelines, or **dataset subclassing** for isolated, reusable components.
- All approaches require minimal code changes—either editing [`clean.py`](https://github.com/fishaudio/fish-speech/blob/main/clean.py), redirecting imports in [`semantic.py`](https://github.com/fishaudio/fish-speech/blob/main/semantic.py), or overriding the `augment` method in a subclass.

## Frequently Asked Questions

### Where is the default text cleaning function located in Fish-Speech?

The default cleaning logic resides in [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py) inside the `clean_text` function. This file contains the `SYMBOLS_MAPPING` dictionary, regex patterns for emoji removal, and the logic for collapsing repeated punctuation.

### Can I use multiple custom preprocessing pipelines simultaneously?

Yes, but you must manage them through separate dataset subclasses or wrapper modules. Since the global import of `clean_text` is shared across the library, running multiple pipelines in the same process requires either subclassing different dataset variants or dynamically swapping the imported function before instantiating each dataset.

### Will modifying clean_text affect inference as well as training?

Yes. The `clean_text` function is used consistently across both training datasets and inference preprocessing paths. Any changes to [`fish_speech/text/clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/clean.py) will automatically apply to real-time text-to-speech generation, ensuring consistency between training data and production inputs.

### How do I preserve the original cleaning behavior while adding new steps?

Create a wrapper function in a new module (e.g., [`fish_speech/text/custom_clean.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/text/custom_clean.py)) that imports the original `clean_text`, applies the base cleaning, and then runs your additional transformations. Redirect the dataset imports to use your wrapper as an alias for `clean_text`, leaving the original file untouched.