# How WaveRNN Vocoder Generates Raw Audio from Mel Spectrograms: Autoregressive Pipeline Explained

> Understand how the WaveRNN vocoder generates raw audio from mel spectrograms. Explore its three-stage autoregressive pipeline for realistic sound synthesis.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: deep-dive
- Published: 2026-03-05

---

**The WaveRNN vocoder converts mel spectrograms into time-domain waveforms through a three-stage autoregressive process involving feature upsampling, dual GRU processing, and categorical sampling over discrete amplitude levels.**

The Real-Time-Voice-Cloning repository by CorentinJ implements a neural vocoder based on the WaveRNN architecture to synthesize high-fidelity speech from mel spectrograms. Understanding how the WaveRNN vocoder generates raw audio from mel spectrograms requires examining the **autoregressive generation loop** implemented in [`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py), which processes conditioning features sample-by-sample to reconstruct the final waveform.

## The Three-Stage Generation Pipeline

The conversion from spectral to temporal domain follows a strict pipeline operating at the audio sample rate.

### Stage 1: Upsampling Conditioning Features

The input mel-spectrogram operates at a lower temporal resolution than the target audio. The `UpsampleNetwork` class (defined around line 60 in [`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py)) handles this mismatch by upsampling the low-rate mel features to the audio sample rate.

This network produces two critical outputs:

- A per-sample conditioning tensor `mels` with shape **batch × time × features**
- An auxiliary tensor `aux` containing learned residual streams that provide fine-grained spectral context

These tensors ensure that every audio sample receives appropriate spectral conditioning during generation.

### Stage 2: Autoregressive RNN Inference

The core generation logic resides in the `WaveRNN.generate` method (starting around line 53). For each timestep, the model concatenates three inputs:

1. The previous audio sample (`x`)
2. The current upsampled mel slice (`m_t`)
3. Four auxiliary slices (`a1_t` through `a4_t`)

This concatenated vector passes through a projection layer (`self.I`) before entering two **GRU cells** (`rnn1` and `rnn2`). The implementation uses `self.get_gru_cell` to extract lightweight `GRUCell` copies from the trained layers, enabling efficient single-step inference without the overhead of full-sequence GRU computation.

Skip connections inject the residual signal throughout the network. The hidden states then flow through two fully-connected layers (`fc1`, `fc2`) and a final linear projection (`fc3`), outputting logits for the next sample prediction.

### Stage 3: RAW Mode Sample Generation

In **RAW mode**, the output logits represent a categorical distribution over `2**bits` discrete amplitude levels (typically 256 classes for 8-bit audio). The sampling code (lines approximately 222-230) uses `torch.distributions.Categorical` to draw a class index, then maps it to the continuous range [-1, 1] using the transformation:

```python
sample = 2 * distrib.sample().float() / (self.n_classes - 1.) - 1.

```

This sample becomes the input `x` for the next timestep, creating the autoregressive chain that continues until reaching `seq_len`.

## Neural Architecture Components

The WaveRNN implementation employs specific architectural choices to balance quality and inference speed.

**Dual GRU Processing**: The `rnn1` and `rnn2` cells process conditioning information sequentially, with the second GRU receiving the output of the first. This hierarchical processing captures both local acoustic details and long-range temporal dependencies.

**Residual Connections**: The auxiliary features (`aux`) provide skip connections that preserve high-frequency information through the upsampling and generation stages, preventing spectral degradation during the mel-to-audio conversion.

**Projection Layers**: The `self.I` layer handles initial feature projection, while `fc1`, `fc2`, and `fc3` progressively refine the hidden representations into the final output distribution.

## Batched Inference Optimization

For production use, the vocoder supports `batched=True` mode, which parallelizes generation across overlapping windows of the spectrogram. The implementation splits upsampled features into overlapping chunks, processes them simultaneously, and reconstructs the final waveform using `xfade_and_unfold`—a cross-fading technique that eliminates boundary artifacts between chunks.

This approach significantly reduces latency for long utterances while maintaining sample quality identical to single-batch generation.

## Post-Processing Pipeline

After generating the raw audio array, the `generate` method applies several reconstruction steps (lines approximately 245-254):

1. **μ-law decoding**: Reverses the compression applied during training
2. **Pre-emphasis removal**: Applies the inverse filter used in preprocessing
3. **Fade-out**: Applies a short amplitude ramp to eliminate click artifacts at the sequence end

The result is a numpy array of float64 values representing the final time-domain waveform.

## Practical Implementation

Accessing the vocoder through the public API in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) abstracts the complexity:

```python
from vocoder import inference as vocoder_infer
import soundfile as sf

# Load checkpoint

vocoder_infer.load_model('saved_models/vocoder/pretrained.pt')

# Generate from mel spectrogram

import numpy as np
mel = np.load('generated_mel.npy')  # Shape: (num_mels, time)

audio = vocoder_infer.infer_waveform(mel)

# Save output

sf.write('output.wav', audio, samplerate=22050)

```

The `infer_waveform` wrapper handles tensor conversion, normalization, and calls `WaveRNN.generate` internally with appropriate hyperparameters from [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py).

## Summary

- The WaveRNN vocoder generates audio through **autoregressive categorical sampling**, predicting one sample at a time conditioned on previous outputs and upsampled mel features.
- **Feature upsampling** in [`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py) aligns the low-rate mel spectrogram with the audio sample rate while preserving spectral detail through auxiliary residual streams.
- **Dual GRU cells** process conditioning information efficiently using lightweight cell extracts for single-step inference.
- **RAW mode** quantizes the waveform into discrete levels (typically 256 for 8-bit) and samples from this categorical distribution using `torch.distributions.Categorical`.
- **Batched generation** with cross-fading enables parallel processing of long sequences without quality degradation.
- Post-processing applies μ-law decoding and pre-emphasis removal to produce the final float64 waveform.

## Frequently Asked Questions

### How does WaveRNN handle the different sampling rates between mel spectrograms and raw audio?

The `UpsampleNetwork` class interpolates the low-temporal-resolution mel features to match the audio sample rate (typically 22,050 Hz), generating per-sample conditioning vectors that guide each autoregressive step.

### What is the difference between RAW mode and other generation modes in WaveRNN?

RAW mode treats audio generation as a **categorical classification problem** over discrete amplitude levels (e.g., 256 values for 8-bit), sampling from a softmax distribution at each step. Other modes like Mixture of Logistics (MOL) model the distribution as a mixture of logistic distributions, though the Real-Time-Voice-Cloning implementation primarily uses RAW mode for simplicity and stability.

### Why does the generation process require the previous sample as input?

WaveRNN is **autoregressive**, meaning each predicted sample conditions on all previously generated samples to maintain temporal consistency and natural-sounding phase continuity. This sequential dependency ensures that the acoustic characteristics evolve smoothly throughout the utterance.

### How does batched generation maintain audio quality across chunk boundaries?

The implementation uses `xfade_and_unfold`, which processes overlapping windows and applies a **cross-fade** between adjacent segments. This technique smooths the amplitude transition at chunk boundaries, preventing audible discontinuities while enabling parallel computation.