How WaveRNN Vocoder Generates Raw Audio from Mel Spectrograms: Autoregressive Pipeline Explained

The WaveRNN vocoder converts mel spectrograms into time-domain waveforms through a three-stage autoregressive process involving feature upsampling, dual GRU processing, and categorical sampling over discrete amplitude levels.

The Real-Time-Voice-Cloning repository by CorentinJ implements a neural vocoder based on the WaveRNN architecture to synthesize high-fidelity speech from mel spectrograms. Understanding how the WaveRNN vocoder generates raw audio from mel spectrograms requires examining the autoregressive generation loop implemented in vocoder/models/fatchord_version.py, which processes conditioning features sample-by-sample to reconstruct the final waveform.

The Three-Stage Generation Pipeline

The conversion from spectral to temporal domain follows a strict pipeline operating at the audio sample rate.

Stage 1: Upsampling Conditioning Features

The input mel-spectrogram operates at a lower temporal resolution than the target audio. The UpsampleNetwork class (defined around line 60 in vocoder/models/fatchord_version.py) handles this mismatch by upsampling the low-rate mel features to the audio sample rate.

This network produces two critical outputs:

• A per-sample conditioning tensor mels with shape batch × time × features
• An auxiliary tensor aux containing learned residual streams that provide fine-grained spectral context

These tensors ensure that every audio sample receives appropriate spectral conditioning during generation.

Stage 2: Autoregressive RNN Inference

The core generation logic resides in the WaveRNN.generate method (starting around line 53). For each timestep, the model concatenates three inputs:

1. The previous audio sample (x)
2. The current upsampled mel slice (m_t)
3. Four auxiliary slices (a1_t through a4_t)

This concatenated vector passes through a projection layer (self.I) before entering two GRU cells (rnn1 and rnn2). The implementation uses self.get_gru_cell to extract lightweight GRUCell copies from the trained layers, enabling efficient single-step inference without the overhead of full-sequence GRU computation.

Skip connections inject the residual signal throughout the network. The hidden states then flow through two fully-connected layers (fc1, fc2) and a final linear projection (fc3), outputting logits for the next sample prediction.

Stage 3: RAW Mode Sample Generation

In RAW mode, the output logits represent a categorical distribution over 2**bits discrete amplitude levels (typically 256 classes for 8-bit audio). The sampling code (lines approximately 222-230) uses torch.distributions.Categorical to draw a class index, then maps it to the continuous range [-1, 1] using the transformation:

sample = 2 * distrib.sample().float() / (self.n_classes - 1.) - 1.

This sample becomes the input x for the next timestep, creating the autoregressive chain that continues until reaching seq_len.

Neural Architecture Components

The WaveRNN implementation employs specific architectural choices to balance quality and inference speed.

Dual GRU Processing: The rnn1 and rnn2 cells process conditioning information sequentially, with the second GRU receiving the output of the first. This hierarchical processing captures both local acoustic details and long-range temporal dependencies.

Residual Connections: The auxiliary features (aux) provide skip connections that preserve high-frequency information through the upsampling and generation stages, preventing spectral degradation during the mel-to-audio conversion.

Projection Layers: The self.I layer handles initial feature projection, while fc1, fc2, and fc3 progressively refine the hidden representations into the final output distribution.

Batched Inference Optimization

For production use, the vocoder supports batched=True mode, which parallelizes generation across overlapping windows of the spectrogram. The implementation splits upsampled features into overlapping chunks, processes them simultaneously, and reconstructs the final waveform using xfade_and_unfold—a cross-fading technique that eliminates boundary artifacts between chunks.

This approach significantly reduces latency for long utterances while maintaining sample quality identical to single-batch generation.

Post-Processing Pipeline

After generating the raw audio array, the generate method applies several reconstruction steps (lines approximately 245-254):

1. μ-law decoding: Reverses the compression applied during training
2. Pre-emphasis removal: Applies the inverse filter used in preprocessing
3. Fade-out: Applies a short amplitude ramp to eliminate click artifacts at the sequence end

The result is a numpy array of float64 values representing the final time-domain waveform.

Practical Implementation

Accessing the vocoder through the public API in vocoder/inference.py abstracts the complexity:

from vocoder import inference as vocoder_infer
import soundfile as sf

# Load checkpoint

vocoder_infer.load_model('saved_models/vocoder/pretrained.pt')

# Generate from mel spectrogram

import numpy as np
mel = np.load('generated_mel.npy')  # Shape: (num_mels, time)

audio = vocoder_infer.infer_waveform(mel)

# Save output

sf.write('output.wav', audio, samplerate=22050)

The infer_waveform wrapper handles tensor conversion, normalization, and calls WaveRNN.generate internally with appropriate hyperparameters from vocoder/hparams.py.

Summary

• The WaveRNN vocoder generates audio through autoregressive categorical sampling, predicting one sample at a time conditioned on previous outputs and upsampled mel features.
• Feature upsampling in vocoder/models/fatchord_version.py aligns the low-rate mel spectrogram with the audio sample rate while preserving spectral detail through auxiliary residual streams.
• Dual GRU cells process conditioning information efficiently using lightweight cell extracts for single-step inference.
• RAW mode quantizes the waveform into discrete levels (typically 256 for 8-bit) and samples from this categorical distribution using torch.distributions.Categorical.
• Batched generation with cross-fading enables parallel processing of long sequences without quality degradation.
• Post-processing applies μ-law decoding and pre-emphasis removal to produce the final float64 waveform.

Frequently Asked Questions

How does WaveRNN handle the different sampling rates between mel spectrograms and raw audio?

The UpsampleNetwork class interpolates the low-temporal-resolution mel features to match the audio sample rate (typically 22,050 Hz), generating per-sample conditioning vectors that guide each autoregressive step.

What is the difference between RAW mode and other generation modes in WaveRNN?

RAW mode treats audio generation as a categorical classification problem over discrete amplitude levels (e.g., 256 values for 8-bit), sampling from a softmax distribution at each step. Other modes like Mixture of Logistics (MOL) model the distribution as a mixture of logistic distributions, though the Real-Time-Voice-Cloning implementation primarily uses RAW mode for simplicity and stability.

Why does the generation process require the previous sample as input?

WaveRNN is autoregressive, meaning each predicted sample conditions on all previously generated samples to maintain temporal consistency and natural-sounding phase continuity. This sequential dependency ensures that the acoustic characteristics evolve smoothly throughout the utterance.

How does batched generation maintain audio quality across chunk boundaries?

The implementation uses xfade_and_unfold, which processes overlapping windows and applies a cross-fade between adjacent segments. This technique smooths the amplitude transition at chunk boundaries, preventing audible discontinuities while enabling parallel computation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}