How to Quantize the Fish-Speech Model for Efficient Inference

Fish-Speech supports INT8 and INT4 weight-only quantization via tools/llama/quantize.py, reducing model size by 4× to 8× while maintaining generation quality.

Quantizing the model for efficient inference allows you to deploy Fish-Speech on resource-constrained hardware without sacrificing audio quality. The fishaudio/fish-speech repository provides built-in tooling for post-training quantization that works with any checkpoint, including the official fish-speech-1.4 release.

Quantization Modes Available in Fish-Speech

Fish-Speech implements two weight-only quantization schemes in tools/llama/quantize.py. These methods compress only the linear layer weights, leaving activations in higher precision (typically BFloat16) to preserve output quality.

Mode	Bit-Width	Quantization Strategy	Typical Size Reduction
INT8	8 bits	Symmetric per-channel	~4× smaller
INT4	4 bits	Group-wise (default groupsize=128)	~8× smaller

The INT8 mode uses symmetric per-channel quantization via dynamically_quantize_per_channel, storing INT8 weights with per-channel scale tensors. The INT4 mode applies group-wise quantization via group_quantize_tensor, packing weights into a custom INT4 format compatible with optimized Llama kernels.

How Model Quantization Works Under the Hood

The quantization system relies on two handler classes that transform nn.Linear layers into compressed representations.

WeightOnlyInt8QuantHandler scans the model architecture for linear layers, computes per-channel scales, and stores INT8 weights alongside their quantization parameters. At runtime, WeightOnlyInt8QuantHandler.convert_for_runtime replaces these with WeightOnlyInt8Linear modules that perform efficient INT8 matrix multiplication.

WeightOnlyInt4QuantHandler operates similarly but uses group-wise quantization with a configurable groupsize parameter (default 128). It packs 4-bit weights into the torch.ops.aten._convert_weight_to_int4pack format. When loading a quantized checkpoint, WeightOnlyInt4QuantHandler.convert_for_runtime injects WeightOnlyInt4Linear layers that execute optimized INT4 kernels.

The runtime conversion logic resides in fish_speech/models/text2semantic/llama.py, which automatically detects quantized tensors during load_model and applies the appropriate handler.

Step-by-Step Guide to Quantize Your Checkpoint

Running the Quantization Script

Use the CLI entry point in tools/llama/quantize.py to convert an existing checkpoint. The script accepts the checkpoint path, quantization mode, and optional parameters for INT4 group size.

# From the repository root

python -m tools.llama.quantize \
    --checkpoint-path checkpoints/fish-speech-1.4 \
    --mode int4 \
    --groupsize 128 \
    --timestamp $(date +%Y%m%d_%H%M%S)

For INT8 quantization, change --mode to int8 and omit the --groupsize parameter:

python -m tools.llama.quantize \
    --checkpoint-path checkpoints/fish-speech-1.4 \
    --mode int8 \
    --timestamp $(date +%Y%m%d_%H%M%S)

Understanding the Output Checkpoint Structure

The quantization script creates a new directory under checkpoints/ with a descriptive suffix indicating the quantization configuration. The naming convention includes the model version, quantization mode, group size (for INT4), and timestamp.

Example output structure:

checkpoints/
├── fish-speech-1.4/              # Original full-precision checkpoint

└── fs-1.4-int4-g128-20260312_154530/
    ├── model.pth                 # Quantized weights (INT4 packed)

    ├── config.json               # Model configuration

    └── tokenizer.json            # Tokenizer assets

The model.pth file contains only the quantized tensors, while non-quantized components (such as VQ-GAN weights) are copied from the source checkpoint.

Loading and Running Inference with Quantized Models

Load a quantized checkpoint using the same load_model function as full-precision models. The detection and conversion of quantized layers happens automatically in fish_speech/models/text2semantic/llama.py.

import torch
from fish_speech.models.text2semantic.llama import load_model

# Load quantized model (INT4 example)

model, decode_one_token = load_model(
    checkpoint_path="checkpoints/fs-1.4-int4-g128-20260312_154530",
    device="cuda",
    precision=torch.bfloat16,
    compile=False,
)

model.eval()

# The model now contains WeightOnlyInt4Linear layers

print(f"Model loaded with {sum(p.numel() for p in model.parameters())} parameters")

For INT8 checkpoints, the process is identical—the runtime automatically instantiates WeightOnlyInt8Linear modules based on the checkpoint contents.

You can then use the model for text-to-semantic generation using the standard generation API:

from fish_speech.inference_engine.utils import encode_text

# Encode prompt

prompt_text = "<|speaker:0|>Hello world!"
prompt_ids = model.tokenizer.encode(prompt_text, add_special_tokens=False)
prompt_tensor = torch.tensor([prompt_ids], dtype=torch.long, device="cuda")

# Generate semantic tokens

output = model.generate(
    prompt=prompt_tensor,
    max_new_tokens=200,
    temperature=0.9,
    top_p=0.95,
    top_k=40,
)

print("Generated shape:", output.shape)

The fish_speech/models/text2semantic/inference.py utilities work unchanged with quantized models, as the quantization affects only the linear layer weights and not the model interface.

Key Source Files and Implementation Details

File	Role	Key Components
tools/llama/quantize.py	CLI entry point and quantization handlers	WeightOnlyInt8QuantHandler, WeightOnlyInt4QuantHandler, dynamically_quantize_per_channel, group_quantize_tensor
fish_speech/models/text2semantic/llama.py	Model definition and runtime conversion	load_model, WeightOnlyInt8Linear, WeightOnlyInt4Linear, automatic handler detection
fish_speech/models/text2semantic/inference.py	High-level generation utilities	generate, text encoding, sampling functions (works with quantized models)

The quantization implementation leverages PyTorch's torch.ops.aten._convert_weight_to_int4pack for efficient INT4 packing and uses custom kernels for fast quantized matrix multiplication during inference.

Summary

• Fish-Speech provides INT8 (8-bit per-channel) and INT4 (4-bit group-wise) weight-only quantization via tools/llama/quantize.py.
• Quantization reduces model size by 4× (INT8) or 8× (INT4) while maintaining generation quality.
• The CLI tool creates new checkpoints with descriptive names (e.g., fs-1.4-int4-g128-<timestamp>) containing packed quantized weights.
• fish_speech/models/text2semantic/llama.py automatically detects quantized checkpoints and swaps in WeightOnlyInt8Linear or WeightOnlyInt4Linear layers at runtime.
• Inference uses the identical API as full-precision models; no code changes are required beyond pointing to the quantized checkpoint path.

Frequently Asked Questions

What is the difference between INT8 and INT4 quantization in Fish-Speech?

INT8 quantization uses symmetric per-channel 8-bit weights, reducing model size by approximately 4×. INT4 quantization uses group-wise 4-bit weights with a default group size of 128, achieving approximately 8× compression. INT4 provides higher compression but may require careful evaluation for specific audio quality requirements, while INT8 offers a balanced trade-off between size and fidelity.

Can I quantize a custom-trained Fish-Speech checkpoint?

Yes, the quantization tools in tools/llama/quantize.py work with any valid Fish-Speech checkpoint, including custom-trained models. Simply specify the path to your checkpoint directory (containing model.pth and config files) using the --checkpoint-path argument. The quantization process preserves your model's architecture while compressing only the linear layer weights.

How do I load and use a quantized model for inference?

Load quantized models using the standard load_model function from fish_speech/models/text2semantic/llama.py. Pass the path to the quantized checkpoint directory (e.g., checkpoints/fs-1.4-int4-g128-20260312_154530). The function automatically detects the quantized format and instantiates the appropriate WeightOnlyInt8Linear or WeightOnlyInt4Linear layers. You can then use model.generate() exactly as you would with a full-precision model.

Does quantization affect the audio quality of generated speech?

Weight-only quantization in Fish-Speech is designed to minimize quality degradation. INT8 quantization typically preserves perceptual quality indistinguishable from the full-precision model for most use cases. INT4 quantization may introduce minor artifacts in some edge cases but generally maintains high fidelity, especially with the default group size of 128. For production deployments where maximum quality is critical, INT8 is recommended; for resource-constrained environments, INT4 offers an excellent compression-to-quality ratio.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}