How the RVQ Audio Codec Encodes and Decodes Speech in Fish-Speech

The RVQ audio codec compresses speech into discrete codebook indices by down-sampling encoder latents, applying hierarchical semantic and residual quantization, and up-sampling the combined representation, while decoding reverses this pipeline to reconstruct waveforms from indices.

The RVQ (Residual Vector Quantization) audio codec serves as the neural audio compression backbone of the Fish-Speech text-to-speech system. According to the fishaudio/fish-speech source code, this codec transforms raw waveforms into compact discrete representations through a sophisticated pipeline of causal convolutions and hierarchical quantization. Understanding how this RVQ audio codec encodes and decodes speech reveals how Fish-Speech achieves high-quality synthesis at extremely low bitrates.

How the RVQ Audio Codec Encodes Speech

The encoding process begins in DAC.encode within fish_speech/models/dac/modded_dac.py, where raw audio undergoes neural compression through a series of causal convolutions and quantization stages before producing discrete indices.

Step 1: Latent Generation and Down-Sampling

First, the encoder produces a continuous latent representation. In modded_dac.py at lines 155‑157, the raw waveform passes through causal convolutional layers:

z = self.encoder(audio_data)

This yields a tensor z of shape [B, D, T] (batch, dimension, time). The DownsampleResidualVectorQuantize module in fish_speech/models/dac/rvq.py then reduces temporal resolution. At lines 300‑301, a down-sampling network applies causal convolutions with a default factor of [2, 2]:

z = self.downsample(z)

An optional pre-processing module (identity by default) may be applied at lines 301‑302.

Step 2: Semantic and Residual Quantization

The core compression happens through two-stage quantization within the forward method of DownsampleResidualVectorQuantize.

Semantic quantization extracts coarse features using a single codebook. At lines 303‑309 in rvq.py, the semantic quantizer processes the down-sampled latent:

semantic_z, semantic_codes, _, _ = self.semantic_quantizer(z)

Residual quantization handles fine-grained details from the reconstruction error. Lines 309‑312 compute the residual and pass it through the multi-codebook quantizer:

residual_z = z - semantic_z
residual_z, codes, _, _ = self.quantizer(residual_z, n_quantizers=n_quantizers)

Step 3: Latent Reconstruction and Output

The codec combines both representations at lines 313‑314 by summing the latents:

z = semantic_z + residual_z

The concatenated indices (torch.cat([semantic_codes, codes], dim=1)) form the final discrete representation of shape [B, N, T'] where N = 1 + n_codebooks (default 10). The latent is up-sampled back to original resolution at lines 318‑319:

z = self.upsample(z)

Lines 325‑334 crop or pad to correct length mismatches, and lines 335‑343 return a VQResult containing the indices and final latent. This achieves approximately 21 Hz frame-rate compression.

How the RVQ Audio Codec Decodes Speech

Decoding reverses the quantization pipeline to reconstruct waveforms from discrete indices. The process occurs in DownsampleResidualVectorQuantize.decode and DAC.from_indices.

Reconstructing from Discrete Indices

The input indices tensor is split at lines 354‑357 in rvq.py. The first column represents semantic codes, while remaining columns contain residual codes:

indices[:, 0] = torch.clamp(indices[:, 0], max=self.semantic_quantizer.n_codebooks-1)

Semantic reconstruction happens at line 361:

z_q_semantic = self.semantic_quantizer.from_codes(indices[:, :1])[0]

Residual reconstruction follows at lines 362‑363:

z_q_residual = self.quantizer.from_codes(indices[:, 1:])[0]

Latent Summation and Waveform Generation

The full latent is recovered by summing both components at line 363:

z_q = z_q_semantic + z_q_residual

After optional post-processing (line 364) and up-sampling (lines 365‑366), the latent returns to its original temporal resolution. In modded_dac.py at lines 221‑223, the from_indices method completes the pipeline:

z = self.quantizer.decode(indices)
return self.decoder(z)

The resulting tensor of shape [B, D, T] can be converted directly into audible speech waveforms.

Practical Implementation Example

Here is how to use the RVQ audio codec within the Fish-Speech framework:

import torch
from fish_speech.models.dac.modded_dac import DAC
from fish_speech.models.dac.rvq import DownsampleResidualVectorQuantize

# Initialize the RVQ quantizer with 8 residual codebooks and 2x2 down-sampling

rvq = DownsampleResidualVectorQuantize(
    input_dim=512,
    n_codebooks=8,
    codebook_dim=8,
    codebook_size=1024,
    quantizer_dropout=0.5,
    downsample_factor=[2, 2],
)

# Build the complete DAC model

model = DAC(
    encoder_dim=64,
    encoder_rates=[2, 4, 8, 8],
    latent_dim=512,
    quantizer=rvq,
    causal=True,
)

# Example: encode 1 second of 44.1kHz audio

audio = torch.randn(1, 44100)
indices, lengths = model.encode(audio)  # Returns discrete RVQ codes [B, N, T']

# Decode back to waveform

reconstructed = model.from_indices(indices)  # Shape: [B, 1, T]

The encode method runs the encoder and RVQ forward pass, while from_indices invokes the RVQ decoder and DAC decoder to reconstruct audio.

Key Source Files and Architecture

Understanding the RVQ audio codec implementation requires familiarity with these specific files in the fishaudio/fish-speech repository:

• fish_speech/models/dac/rvq.py: Contains the DownsampleResidualVectorQuantize class implementing down-sampling, semantic and residual quantization, and the decode routine.
• fish_speech/models/dac/modded_dac.py: Houses the high-level DAC model that orchestrates the encoder, RVQ quantizer, and decoder, providing the encode() and from_indices() APIs.
• fish_speech/models/dac/encoder.py: Implements the causal convolutional encoder that produces continuous latents fed into the RVQ module.
• fish_speech/models/dac/decoder.py: Provides the transposed convolutional decoder that converts RVQ-decoded latents back into time-domain waveforms.

Summary

The RVQ audio codec in Fish-Speech achieves efficient speech compression through:

• Hierarchical quantization separating coarse semantic features from fine residual details using distinct codebooks in rvq.py.
• Temporal compression via causal down-sampling convolutions that reduce frame rates to approximately 21 Hz before quantization.
• Additive reconstruction combining semantic and residual latents during both encoding (lines 313‑314) and decoding (line 363) phases.
• End-to-end integration with the DAC encoder/decoder architecture defined in modded_dac.py, enabling seamless conversion between waveforms and discrete indices.

This architecture enables high-fidelity speech representation at bitrates significantly lower than traditional audio codecs.

Frequently Asked Questions

What is the difference between semantic and residual quantization in the RVQ codec?

Semantic quantization uses a single codebook to capture coarse, high-level speech features, while residual quantization employs multiple codebooks (default 9) to encode fine-grained acoustic details from the difference between the original and semantic latents. According to rvq.py lines 303‑312, these operate sequentially with the residual quantizer processing z - semantic_z.

How does the down-sampling factor affect the RVQ audio codec bitrate?

The downsample_factor parameter (default [2, 2] at lines 300‑301 in rvq.py) reduces the temporal resolution before quantization, directly determining the final frame rate of approximately 21 Hz. Increasing these factors would further compress the representation but potentially reduce reconstruction quality.

Can the RVQ codec be used independently of the Fish-Speech model?

Yes. As shown in the implementation example, DownsampleResidualVectorQuantize can be instantiated separately from the full DAC model, though it requires compatible encoder outputs. The codec operates as a standalone neural audio compressor optimized for the Fish-Speech latent space.

What are the default quantization parameters for the RVQ audio codec?

The default configuration uses 10 total codebooks (1 semantic + 9 residual), a codebook size of 1024, and 8-dimensional codebook vectors. These defaults are defined in the DownsampleResidualVectorQuantize constructor within rvq.py and balance compression efficiency with reconstruction fidelity at the target frame rate.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}