# How the RVQ Audio Codec Encodes and Decodes Speech in Fish-Speech

> Discover how the RVQ audio codec encodes speech by quantizing latents and reconstructs waveforms from indices, offering efficient compression in Fish-Speech.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: deep-dive
- Published: 2026-03-12

---

**The RVQ audio codec compresses speech into discrete codebook indices by down-sampling encoder latents, applying hierarchical semantic and residual quantization, and up-sampling the combined representation, while decoding reverses this pipeline to reconstruct waveforms from indices.**

The RVQ (Residual Vector Quantization) audio codec serves as the neural audio compression backbone of the **Fish-Speech** text-to-speech system. According to the `fishaudio/fish-speech` source code, this codec transforms raw waveforms into compact discrete representations through a sophisticated pipeline of causal convolutions and hierarchical quantization. Understanding how this RVQ audio codec encodes and decodes speech reveals how Fish-Speech achieves high-quality synthesis at extremely low bitrates.

## How the RVQ Audio Codec Encodes Speech

The encoding process begins in `DAC.encode` within **[`fish_speech/models/dac/modded_dac.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/modded_dac.py)**, where raw audio undergoes neural compression through a series of causal convolutions and quantization stages before producing discrete indices.

### Step 1: Latent Generation and Down-Sampling

First, the encoder produces a continuous latent representation. In **[`modded_dac.py`](https://github.com/fishaudio/fish-speech/blob/main/modded_dac.py)** at lines 155‑157, the raw waveform passes through causal convolutional layers:

```python
z = self.encoder(audio_data)

```

This yields a tensor `z` of shape `[B, D, T]` (batch, dimension, time). The `DownsampleResidualVectorQuantize` module in **[`fish_speech/models/dac/rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/rvq.py)** then reduces temporal resolution. At lines 300‑301, a down-sampling network applies causal convolutions with a default factor of `[2, 2]`:

```python
z = self.downsample(z)

```

An optional pre-processing module (identity by default) may be applied at lines 301‑302.

### Step 2: Semantic and Residual Quantization

The core compression happens through two-stage quantization within the `forward` method of `DownsampleResidualVectorQuantize`.

**Semantic quantization** extracts coarse features using a single codebook. At lines 303‑309 in **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)**, the semantic quantizer processes the down-sampled latent:

```python
semantic_z, semantic_codes, _, _ = self.semantic_quantizer(z)

```

**Residual quantization** handles fine-grained details from the reconstruction error. Lines 309‑312 compute the residual and pass it through the multi-codebook quantizer:

```python
residual_z = z - semantic_z
residual_z, codes, _, _ = self.quantizer(residual_z, n_quantizers=n_quantizers)

```

### Step 3: Latent Reconstruction and Output

The codec combines both representations at lines 313‑314 by summing the latents:

```python
z = semantic_z + residual_z

```

The concatenated indices (`torch.cat([semantic_codes, codes], dim=1)`) form the final discrete representation of shape `[B, N, T']` where `N = 1 + n_codebooks` (default 10). The latent is up-sampled back to original resolution at lines 318‑319:

```python
z = self.upsample(z)

```

Lines 325‑334 crop or pad to correct length mismatches, and lines 335‑343 return a `VQResult` containing the indices and final latent. This achieves approximately 21 Hz frame-rate compression.

## How the RVQ Audio Codec Decodes Speech

Decoding reverses the quantization pipeline to reconstruct waveforms from discrete indices. The process occurs in `DownsampleResidualVectorQuantize.decode` and `DAC.from_indices`.

### Reconstructing from Discrete Indices

The input indices tensor is split at lines 354‑357 in **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)**. The first column represents semantic codes, while remaining columns contain residual codes:

```python
indices[:, 0] = torch.clamp(indices[:, 0], max=self.semantic_quantizer.n_codebooks-1)

```

Semantic reconstruction happens at line 361:

```python
z_q_semantic = self.semantic_quantizer.from_codes(indices[:, :1])[0]

```

Residual reconstruction follows at lines 362‑363:

```python
z_q_residual = self.quantizer.from_codes(indices[:, 1:])[0]

```

### Latent Summation and Waveform Generation

The full latent is recovered by summing both components at line 363:

```python
z_q = z_q_semantic + z_q_residual

```

After optional post-processing (line 364) and up-sampling (lines 365‑366), the latent returns to its original temporal resolution. In **[`modded_dac.py`](https://github.com/fishaudio/fish-speech/blob/main/modded_dac.py)** at lines 221‑223, the `from_indices` method completes the pipeline:

```python
z = self.quantizer.decode(indices)
return self.decoder(z)

```

The resulting tensor of shape `[B, D, T]` can be converted directly into audible speech waveforms.

## Practical Implementation Example

Here is how to use the RVQ audio codec within the Fish-Speech framework:

```python
import torch
from fish_speech.models.dac.modded_dac import DAC
from fish_speech.models.dac.rvq import DownsampleResidualVectorQuantize

# Initialize the RVQ quantizer with 8 residual codebooks and 2x2 down-sampling

rvq = DownsampleResidualVectorQuantize(
    input_dim=512,
    n_codebooks=8,
    codebook_dim=8,
    codebook_size=1024,
    quantizer_dropout=0.5,
    downsample_factor=[2, 2],
)

# Build the complete DAC model

model = DAC(
    encoder_dim=64,
    encoder_rates=[2, 4, 8, 8],
    latent_dim=512,
    quantizer=rvq,
    causal=True,
)

# Example: encode 1 second of 44.1kHz audio

audio = torch.randn(1, 44100)
indices, lengths = model.encode(audio)  # Returns discrete RVQ codes [B, N, T']

# Decode back to waveform

reconstructed = model.from_indices(indices)  # Shape: [B, 1, T]

```

The `encode` method runs the encoder and RVQ forward pass, while `from_indices` invokes the RVQ decoder and DAC decoder to reconstruct audio.

## Key Source Files and Architecture

Understanding the RVQ audio codec implementation requires familiarity with these specific files in the `fishaudio/fish-speech` repository:

- **[`fish_speech/models/dac/rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/rvq.py)**: Contains the `DownsampleResidualVectorQuantize` class implementing down-sampling, semantic and residual quantization, and the `decode` routine.
- **[`fish_speech/models/dac/modded_dac.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/modded_dac.py)**: Houses the high-level `DAC` model that orchestrates the encoder, RVQ quantizer, and decoder, providing the `encode()` and `from_indices()` APIs.
- **[`fish_speech/models/dac/encoder.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/encoder.py)**: Implements the causal convolutional encoder that produces continuous latents fed into the RVQ module.
- **[`fish_speech/models/dac/decoder.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/models/dac/decoder.py)**: Provides the transposed convolutional decoder that converts RVQ-decoded latents back into time-domain waveforms.

## Summary

The RVQ audio codec in Fish-Speech achieves efficient speech compression through:

- **Hierarchical quantization** separating coarse semantic features from fine residual details using distinct codebooks in **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)**.
- **Temporal compression** via causal down-sampling convolutions that reduce frame rates to approximately 21 Hz before quantization.
- **Additive reconstruction** combining semantic and residual latents during both encoding (lines 313‑314) and decoding (line 363) phases.
- **End-to-end integration** with the DAC encoder/decoder architecture defined in **[`modded_dac.py`](https://github.com/fishaudio/fish-speech/blob/main/modded_dac.py)**, enabling seamless conversion between waveforms and discrete indices.

This architecture enables high-fidelity speech representation at bitrates significantly lower than traditional audio codecs.

## Frequently Asked Questions

### What is the difference between semantic and residual quantization in the RVQ codec?

**Semantic quantization** uses a single codebook to capture coarse, high-level speech features, while **residual quantization** employs multiple codebooks (default 9) to encode fine-grained acoustic details from the difference between the original and semantic latents. According to **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)** lines 303‑312, these operate sequentially with the residual quantizer processing `z - semantic_z`.

### How does the down-sampling factor affect the RVQ audio codec bitrate?

The `downsample_factor` parameter (default `[2, 2]` at lines 300‑301 in **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)**) reduces the temporal resolution before quantization, directly determining the final frame rate of approximately 21 Hz. Increasing these factors would further compress the representation but potentially reduce reconstruction quality.

### Can the RVQ codec be used independently of the Fish-Speech model?

Yes. As shown in the implementation example, `DownsampleResidualVectorQuantize` can be instantiated separately from the full `DAC` model, though it requires compatible encoder outputs. The codec operates as a standalone neural audio compressor optimized for the Fish-Speech latent space.

### What are the default quantization parameters for the RVQ audio codec?

The default configuration uses **10 total codebooks** (1 semantic + 9 residual), a **codebook size of 1024**, and **8-dimensional codebook vectors**. These defaults are defined in the `DownsampleResidualVectorQuantize` constructor within **[`rvq.py`](https://github.com/fishaudio/fish-speech/blob/main/rvq.py)** and balance compression efficiency with reconstruction fidelity at the target frame rate.