# How to Quantize the Fish-Speech Model for Efficient Inference

> Learn how to quantize the Fish-Speech model using INT8 and INT4 for efficient inference. Reduce model size by up to 8x while preserving generation quality.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: how-to-guide
- Published: 2026-03-12

---

**Fish-Speech supports INT8 and INT4 weight-only quantization via [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py), reducing model size by 4× to 8× while maintaining generation quality.**

Quantizing the model for efficient inference allows you to deploy Fish-Speech on resource-constrained hardware without sacrificing audio quality. The `fishaudio/fish-speech` repository provides built-in tooling for post-training quantization that works with any checkpoint, including the official `fish-speech-1.4` release.

## Quantization Modes Available in Fish-Speech

Fish-Speech implements two **weight-only** quantization schemes in [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py). These methods compress only the linear layer weights, leaving activations in higher precision (typically BFloat16) to preserve output quality.

| Mode | Bit-Width | Quantization Strategy | Typical Size Reduction |
|------|-----------|----------------------|------------------------|
| **INT8** | 8 bits | Symmetric per-channel | ~4× smaller |
| **INT4** | 4 bits | Group-wise (default `groupsize=128`) | ~8× smaller |

The **INT8** mode uses symmetric per-channel quantization via `dynamically_quantize_per_channel`, storing INT8 weights with per-channel scale tensors. The **INT4** mode applies group-wise quantization via `group_quantize_tensor`, packing weights into a custom INT4 format compatible with optimized Llama kernels.

## How Model Quantization Works Under the Hood

The quantization system relies on two handler classes that transform `nn.Linear` layers into compressed representations.

**`WeightOnlyInt8QuantHandler`** scans the model architecture for linear layers, computes per-channel scales, and stores INT8 weights alongside their quantization parameters. At runtime, `WeightOnlyInt8QuantHandler.convert_for_runtime` replaces these with `WeightOnlyInt8Linear` modules that perform efficient INT8 matrix multiplication.

**`WeightOnlyInt4QuantHandler`** operates similarly but uses group-wise quantization with a configurable `groupsize` parameter (default 128). It packs 4-bit weights into the `torch.ops.aten._convert_weight_to_int4pack` format. When loading a quantized checkpoint, `WeightOnlyInt4QuantHandler.convert_for_runtime` injects `WeightOnlyInt4Linear` layers that execute optimized INT4 kernels.

The runtime conversion logic resides in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py), which automatically detects quantized tensors during `load_model` and applies the appropriate handler.

## Step-by-Step Guide to Quantize Your Checkpoint

### Running the Quantization Script

Use the CLI entry point in [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py) to convert an existing checkpoint. The script accepts the checkpoint path, quantization mode, and optional parameters for INT4 group size.

```bash

# From the repository root

python -m tools.llama.quantize \
    --checkpoint-path checkpoints/fish-speech-1.4 \
    --mode int4 \
    --groupsize 128 \
    --timestamp $(date +%Y%m%d_%H%M%S)

```

For INT8 quantization, change `--mode` to `int8` and omit the `--groupsize` parameter:

```bash
python -m tools.llama.quantize \
    --checkpoint-path checkpoints/fish-speech-1.4 \
    --mode int8 \
    --timestamp $(date +%Y%m%d_%H%M%S)

```

### Understanding the Output Checkpoint Structure

The quantization script creates a new directory under `checkpoints/` with a descriptive suffix indicating the quantization configuration. The naming convention includes the model version, quantization mode, group size (for INT4), and timestamp.

Example output structure:

```

checkpoints/
├── fish-speech-1.4/              # Original full-precision checkpoint

└── fs-1.4-int4-g128-20260312_154530/
    ├── model.pth                 # Quantized weights (INT4 packed)

    ├── config.json               # Model configuration

    └── tokenizer.json            # Tokenizer assets

```

The `model.pth` file contains only the quantized tensors, while non-quantized components (such as VQ-GAN weights) are copied from the source checkpoint.

## Loading and Running Inference with Quantized Models

Load a quantized checkpoint using the same `load_model` function as full-precision models. The detection and conversion of quantized layers happens automatically in [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py).

```python
import torch
from fish_speech.models.text2semantic.llama import load_model

# Load quantized model (INT4 example)

model, decode_one_token = load_model(
    checkpoint_path="checkpoints/fs-1.4-int4-g128-20260312_154530",
    device="cuda",
    precision=torch.bfloat16,
    compile=False,
)

model.eval()

# The model now contains WeightOnlyInt4Linear layers

print(f"Model loaded with {sum(p.numel() for p in model.parameters())} parameters")

```

For INT8 checkpoints, the process is identical—the runtime automatically instantiates `WeightOnlyInt8Linear` modules based on the checkpoint contents.

You can then use the model for text-to-semantic generation using the standard generation API:

```python
from fish_speech.inference_engine.utils import encode_text

# Encode prompt

prompt_text = "<|speaker:0|>Hello world!"
prompt_ids = model.tokenizer.encode(prompt_text, add_special_tokens=False)
prompt_tensor = torch.tensor([prompt_ids], dtype=torch.long, device="cuda")

# Generate semantic tokens

output = model.generate(
    prompt=prompt_tensor,
    max_new_tokens=200,
    temperature=0.9,
    top_p=0.95,
    top_k=40,
)

print("Generated shape:", output.shape)

```

The [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) utilities work unchanged with quantized models, as the quantization affects only the linear layer weights and not the model interface.

## Key Source Files and Implementation Details

| File | Role | Key Components |
|------|------|----------------|
| [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py) | CLI entry point and quantization handlers | `WeightOnlyInt8QuantHandler`, `WeightOnlyInt4QuantHandler`, `dynamically_quantize_per_channel`, `group_quantize_tensor` |
| [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py) | Model definition and runtime conversion | `load_model`, `WeightOnlyInt8Linear`, `WeightOnlyInt4Linear`, automatic handler detection |
| [`fish_speech/models/text2semantic/inference.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/inference.py) | High-level generation utilities | `generate`, text encoding, sampling functions (works with quantized models) |

The quantization implementation leverages PyTorch's `torch.ops.aten._convert_weight_to_int4pack` for efficient INT4 packing and uses custom kernels for fast quantized matrix multiplication during inference.

## Summary

- Fish-Speech provides **INT8** (8-bit per-channel) and **INT4** (4-bit group-wise) weight-only quantization via [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py).
- Quantization reduces model size by **4× (INT8)** or **8× (INT4)** while maintaining generation quality.
- The CLI tool creates new checkpoints with descriptive names (e.g., `fs-1.4-int4-g128-<timestamp>`) containing packed quantized weights.
- [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py) automatically detects quantized checkpoints and swaps in `WeightOnlyInt8Linear` or `WeightOnlyInt4Linear` layers at runtime.
- Inference uses the identical API as full-precision models; no code changes are required beyond pointing to the quantized checkpoint path.

## Frequently Asked Questions

### What is the difference between INT8 and INT4 quantization in Fish-Speech?

**INT8 quantization** uses symmetric per-channel 8-bit weights, reducing model size by approximately 4×. **INT4 quantization** uses group-wise 4-bit weights with a default group size of 128, achieving approximately 8× compression. INT4 provides higher compression but may require careful evaluation for specific audio quality requirements, while INT8 offers a balanced trade-off between size and fidelity.

### Can I quantize a custom-trained Fish-Speech checkpoint?

Yes, the quantization tools in [`tools/llama/quantize.py`](https://github.com/fishaudio/fish-speech/blob/main/tools/llama/quantize.py) work with any valid Fish-Speech checkpoint, including custom-trained models. Simply specify the path to your checkpoint directory (containing `model.pth` and config files) using the `--checkpoint-path` argument. The quantization process preserves your model's architecture while compressing only the linear layer weights.

### How do I load and use a quantized model for inference?

Load quantized models using the standard `load_model` function from [`fish_speech/models/text2semantic/llama.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/text2semantic/llama.py). Pass the path to the quantized checkpoint directory (e.g., `checkpoints/fs-1.4-int4-g128-20260312_154530`). The function automatically detects the quantized format and instantiates the appropriate `WeightOnlyInt8Linear` or `WeightOnlyInt4Linear` layers. You can then use `model.generate()` exactly as you would with a full-precision model.

### Does quantization affect the audio quality of generated speech?

Weight-only quantization in Fish-Speech is designed to minimize quality degradation. INT8 quantization typically preserves perceptual quality indistinguishable from the full-precision model for most use cases. INT4 quantization may introduce minor artifacts in some edge cases but generally maintains high fidelity, especially with the default group size of 128. For production deployments where maximum quality is critical, INT8 is recommended; for resource-constrained environments, INT4 offers an excellent compression-to-quality ratio.