# How VQManager Handles Reference Audio Encoding in Fish Speech

> Discover how VQManager encodes reference audio into discrete prompt tokens via dynamic sample rate detection, torchaudio loading, tensor shaping, and DAC VQ-GAN encoding. Learn the four-stage process.

- Repository: [Fish Audio/fish-speech](https://github.com/fishaudio/fish-speech)
- Tags: internals
- Published: 2026-03-12

---

**VQManager converts raw reference audio files or bytes into discrete prompt tokens through a four-stage pipeline: dynamic sample-rate detection, torchaudio loading with automatic resampling, tensor shaping, and DAC VQ-GAN encoding.**

The `VQManager` class in the fish-speech repository serves as the critical bridge between raw audio inputs and the LLaMA-based language model. Located in [`fish_speech/inference_engine/vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/vq_manager.py), this component orchestrates the transformation of reference audio into the quantized token sequences that condition neural speech synthesis, leveraging multiple inheritance from `ReferenceLoader` to handle both file system and in-memory audio sources.

## The VQManager Encoding Architecture

`VQManager` operates as a mixin within the broader inference engine, combining audio I/O capabilities from `ReferenceLoader` with VQ-GAN encoding logic. When the `TTSInferenceEngine` receives a request containing reference audio—whether as a file path, raw bytes, or a cached reference ID—it delegates token generation to `VQManager.encode_reference()`. This method abstracts the entire preprocessing chain, ensuring that audio conforms to the decoder model's expected sample rate and tensor format before quantization.

## The Four-Stage Encoding Pipeline

### Stage 1: Sample Rate Detection from Decoder Model

Before loading audio, `VQManager` determines the target sample rate by inspecting the decoder model's configuration. According to lines 27-31 in [`fish_speech/inference_engine/vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/vq_manager.py), the code first checks for a `spec_transform` attribute and uses its `sample_rate`; otherwise, it falls back to `decoder_model.sample_rate`:

```python
if hasattr(self.decoder_model, "spec_transform"):
    sample_rate = self.decoder_model.spec_transform.sample_rate
else:
    sample_rate = self.decoder_model.sample_rate

```

This dynamic detection ensures compatibility with different DAC model configurations without hardcoding constants.

### Stage 2: Audio Loading and Resampling

The manager invokes `self.load_audio(reference_audio, sample_rate)`, a method provided by the `ReferenceLoader` mixin (defined in [`fish_speech/inference_engine/reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py)). As implemented in lines 9-27, this method handles both file paths and in-memory `bytes` objects using **torchaudio**. It automatically converts multi-channel audio to mono and resamples when the source rate differs from the target:

```python

# From reference_loader.py lines 17-27

waveform, original_sr = torchaudio.load(audio_source)
if waveform.shape[0] > 1:
    waveform = waveform.mean(dim=0, keepdim=True)
if original_sr != target_sr:
    resampler = torchaudio.transforms.Resample(original_sr, target_sr)
    waveform = resampler(waveform)

```

### Stage 3: Tensor Conversion and Device Placement

Once loaded as a NumPy array, the waveform undergoes tensor conversion. Lines 33-36 in [`vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/vq_manager.py) wrap the data using `torch.from_numpy()`, transfer it to the decoder's compute device, and reshape it to `[batch, channel, time]`—specifically `[1, 1, time]` for single-channel, single-batch inference:

```python
audios = torch.from_numpy(waveform).to(self.decoder_model.device)
audios = audios.unsqueeze(0).unsqueeze(0)  # [1, 1, T]

```

### Stage 4: VQ-GAN Token Extraction

For DAC-based decoders, `VQManager` calls the encoder to produce discrete tokens. Lines 44-47 in [`vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/vq_manager.py) invoke `self.decoder_model.encode(audios, audio_lengths)`, which returns a list of token sequences. The manager extracts the primary prompt tokens by indexing `[0][0]`:

```python
prompt_tokens = self.decoder_model.encode(audios, audio_lengths)[0][0]

```

If `enable_reference_audio` is `False` or the input is empty, the method returns `None` and emits a log entry (lines 49-52), allowing the inference engine to proceed with unconditional generation.

## Integration with TTSInferenceEngine

`VQManager` is mixed into `TTSInferenceEngine` alongside `ReferenceLoader`. When processing requests containing `reference_id` or `ServeReferenceAudio` objects, the engine calls `ReferenceLoader.load_by_id()` or `load_by_hash()`. Both functions internally invoke `self.encode_reference()` to obtain prompt tokens, as shown in lines 52-61 and 85-93 of [`reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/reference_loader.py). These tokens are then concatenated with text tokens and fed into the LLaMA model to condition the generation of speaker-specific speech characteristics.

## Practical Implementation Examples

### Encoding a Single Reference File Manually

For custom pipelines or debugging, you can instantiate `VQManager` directly with a pre-loaded DAC model:

```python
from fish_speech.inference_engine.vq_manager import VQManager
from fish_speech.inference_engine.reference_loader import ReferenceLoader
from fish_speech.models.dac.modded_dac import DAC

# Initialize the DAC decoder

decoder = DAC.from_pretrained("path/to/dac/checkpoint")

# Configure VQManager with decoder and loader

vq = VQManager()
vq.decoder_model = decoder

# Attach the load_audio method from ReferenceLoader

loader = ReferenceLoader()
loader.decoder_model = decoder
vq.load_audio = loader.load_audio

# Encode a WAV file to prompt tokens

prompt_tokens = vq.encode_reference(
    reference_audio="samples/reference.wav",
    enable_reference_audio=True,
)

print(f"Prompt token shape: {prompt_tokens.shape}")
print(f"Token dtype: {prompt_tokens.dtype}")

```

### Processing References via TTSInferenceEngine

For production inference using the high-level API:

```python
from fish_speech.inference_engine import TTSInferenceEngine
from fish_speech.models.dac.modded_dac import DAC
from fish_speech.utils.schema import ServeTTSRequest
import queue
import torch

# Initialize models

decoder = DAC.from_pretrained("path/to/dac/checkpoint")
llama_queue = queue.Queue()

# Create inference engine (VQManager is mixed in automatically)

engine = TTSInferenceEngine(
    llama_queue=llama_queue,
    decoder_model=decoder,
    precision=torch.float16,
    compile=False,
)

# Build request with reference ID

request = ServeTTSRequest(
    text="Hello world, this is a cloned voice.",
    reference_id="speaker_001",
    use_memory_cache="on",
    streaming=False,
)

# Execute inference

for result in engine.inference(request):
    if result.code == "final":
        sample_rate, audio_array = result.audio
        print(f"Generated {len(audio_array)} samples at {sample_rate} Hz")

```

## Summary

- **Dynamic sample-rate detection** inspects `spec_transform.sample_rate` or `decoder_model.sample_rate` to match decoder expectations
- **ReferenceLoader mixin** handles file I/O, mono conversion, and resampling via torchaudio in [`fish_speech/inference_engine/reference_loader.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/reference_loader.py)
- **Tensor shaping** converts NumPy waveforms to `[batch, channel, time]` format (typically `[1, 1, T]`) on the target device
- **VQ-GAN encoding** extracts discrete tokens via `DAC.encode()`, selecting the first sequence at index `[0][0]` for prompt generation
- **Conditional execution** returns `None` when `enable_reference_audio` is disabled, allowing unconditional synthesis

## Frequently Asked Questions

### What audio formats does VQManager support for reference encoding?

`VQManager` supports any format readable by **torchaudio**, including WAV, MP3, FLAC, and OGG. The `ReferenceLoader.load_audio()` method accepts both file paths (strings) and in-memory `bytes` objects, making it compatible with HTTP uploads and cached binary data.

### How does VQManager handle sample rate mismatches?

The component detects the decoder's required sample rate during initialization. If the source audio's rate differs, `ReferenceLoader` automatically instantiates a `torchaudio.transforms.Resample` transform to convert the waveform to the target rate before tensor conversion, ensuring the DAC model receives correctly scaled input.

### Why does the DAC encode method return a list, and why select `[0][0]`?

The `DAC.encode()` method returns a list of codebook entries or token sequences representing different levels of the VQ-GAN hierarchy. For reference audio encoding, `VQManager` requires the primary top-level tokens, which reside at the first position of the first sequence in the returned list, hence the `[0][0]` indexing pattern seen in [`vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/vq_manager.py) lines 44-47.

### Can VQManager process multiple reference audio files in a single batch?

While the tensor preparation code shapes inputs to `[batch, channel, time]`, the current implementation in [`fish_speech/inference_engine/vq_manager.py`](https://github.com/fishaudio/fish-speech/blob/main/fish_speech/inference_engine/vq_manager.py) is optimized for single-reference inference (batch size 1). To process multiple references, you would need to modify the `encode_reference()` method to stack multiple audio tensors along the batch dimension and handle the corresponding list indexing for multiple token sequences.