How to Integrate a Custom-Trained Vocoder with the Real-Time-Voice-Cloning Synthesizer

You can integrate a custom-trained vocoder into the Real-Time-Voice-Cloning pipeline by training a WaveRNN model using the provided scripts, saving the checkpoint, and loading it via vocoder.inference.load_model() before calling infer_waveform() on mel-spectrograms generated by the synthesizer.

The CorentinJ/Real-Time-Voice-Cloning repository implements a modular voice cloning architecture where the synthesizer and vocoder communicate through a standardized mel-spectrogram interface. Because the synthesizer only outputs NumPy arrays representing mel-spectrograms, you can integrate a custom-trained vocoder without modifying the core synthesis code, provided your model respects the shared hyper-parameters. This separation allows you to swap the default WaveRNN with your own trained checkpoint to achieve custom audio characteristics or improved inference speed.

Architecture Overview

The repository organizes voice cloning into three sequential neural components:

• Encoder – Extracts a 256-dimensional speaker embedding from reference audio.
• Synthesizer – A Tacotron-based model that converts text and speaker embeddings into mel-spectrograms. The inference logic resides in synthesizer/inference.py, specifically within the Synthesizer.synthesize_spectrograms method (lines 70‑84).
• Vocoder – A WaveRNN model that transforms mel-spectrograms into raw waveforms. The entry points are load_model and infer_waveform in vocoder/inference.py (lines 8‑46).

The integration point between the synthesizer and vocoder is strictly the mel-spectrogram object (np.ndarray). Any vocoder architecture that accepts this tensor shape and the hyper-parameters defined in vocoder/hparams.py can replace the default implementation.

Step-by-Step Integration Guide

Step 1: Train Your Custom Vocoder

Train your vocoder using the repository’s training scripts. You can use either the high-level wrapper vocoder_train.py or the underlying vocoder/train.py module. The training process saves a checkpoint file named <model_name>.pt containing the model weights and optimizer state.

Place this checkpoint in your preferred directory, such as models/custom/, without modifying the repository structure.

Step 2: Load the Custom Model at Runtime

Before running inference, load your checkpoint by calling vocoder.inference.load_model() with the path to your .pt file. This function replaces the default WaveRNN instance stored in the internal _model variable, as implemented in vocoder/inference.py (lines 8‑15).

from vocoder import inference as vocoder
from pathlib import Path

custom_vocoder_path = Path("models/custom/my_vocoder.pt")
vocoder.load_model(custom_vocoder_path)

Step 3: Align Hyper-Parameters

The vocoder relies on shared hyper-parameters defined in vocoder/hparams.py. If your training modified any settings—such as voc_rnn_dims, bits, or sample_rate—you must ensure these values match between the synthesizer and vocoder. Critical parameters that must align include:

• sample_rate
• hop_length
• num_mels

Edit vocoder/hparams.py directly or import a modified configuration before loading your model to maintain compatibility.

Step 4: Execute the Full Pipeline

Generate audio by first creating mel-spectrograms with the synthesizer, then passing them to your custom vocoder:

1. Instantiate the synthesizer with a trained Tacotron checkpoint using synthesizer/inference.py.
2. Generate mel-spectrograms via synthesizer.synthesize_spectrograms(texts, speaker_embeddings).
3. Convert the spectrograms to waveforms using vocoder.infer_waveform(mel_spec).

Complete Implementation Example

The following example demonstrates loading both the synthesizer and a custom vocoder, then generating audio from text:

# -------------------------------------------------

# 1️⃣  Load the Synthesizer (pre‑trained Tacotron)

# -------------------------------------------------

from pathlib import Path
from synthesizer.inference import Synthesizer

synth_path = Path("saved_models/synthesizer.pt")
synth = Synthesizer(synth_path)

# -------------------------------------------------

# 2️⃣  Generate a mel‑spectrogram for some text

# -------------------------------------------------

texts = ["Hello, this is a custom vocoder test."]
import numpy as np

# Use a zero‑vector speaker embedding for demonstration

speaker_emb = np.zeros((256,))  # Replace with encoder output in production

mel_specs = synth.synthesize_spectrograms(texts, speaker_emb)[0]

# -------------------------------------------------

# 3️⃣  Load your custom‑trained vocoder

# -------------------------------------------------

from vocoder import inference as vocoder

custom_vocoder_path = Path("models/custom/my_vocoder.pt")
vocoder.load_model(custom_vocoder_path)

# -------------------------------------------------

# 4️⃣  Convert the mel‑spectrogram to waveform

# -------------------------------------------------

wav = vocoder.infer_waveform(mel_specs)

# -------------------------------------------------

# 5️⃣  Save or play the waveform

# -------------------------------------------------

import soundfile as sf
sf.write("output.wav", wav.cpu().numpy(), samplerate=22050)  # sample_rate from hparams

Key implementation details from the source code:

• Synthesizer.synthesize_spectrograms returns a list of mel arrays compatible with the vocoder input requirements.
• vocoder.load_model hot-swaps the default WaveRNN with your custom checkpoint.
• vocoder.infer_waveform automatically normalizes the mel-spectrogram using hp.mel_max_abs_value constants defined during training.

Critical Files for Integration

File	Role	Key Components
synthesizer/inference.py	Tacotron-based synthesizer	Synthesizer.synthesize_spectrograms (lines 70‑84)
vocoder/inference.py	Vocoder inference wrapper	load_model, infer_waveform (lines 8‑46)
vocoder/hparams.py	Shared hyper-parameters	sample_rate, hop_length, num_mels, mel_max_abs_value
vocoder/train.py	Training script	Entry point for custom vocoder training
vocoder/models/fatchord_version.py	Reference WaveRNN implementation	Default architecture used by the repository

Summary

• The mel-spectrogram serves as the universal interface between the synthesizer and any custom vocoder.
• Call vocoder.inference.load_model() to replace the default WaveRNN with your trained checkpoint.
• Ensure vocoder/hparams.py values match your training configuration to prevent audio distortion.
• No modifications to synthesizer/inference.py are required; the synthesizer operates independently of the vocoder implementation.

Frequently Asked Questions

Can I use a vocoder architecture other than WaveRNN?

Yes. You can integrate any vocoder architecture—such as HiFi-GAN or Griffin-Lim—provided it accepts a mel-spectrogram with the shape (num_mels, time) and the sample rate specified in vocoder/hparams.py. You must wrap your model with a compatible load_model and infer_waveform interface similar to vocoder/inference.py.

Where exactly does the synthesizer output the mel-spectrogram?

According to the source code in synthesizer/inference.py, the Synthesizer.synthesize_spectrograms method (lines 70‑84) processes input text and speaker embeddings through the Tacotron model and returns a Python list containing NumPy arrays, where each array represents a mel-spectrogram.

Do I need to retrain the synthesizer when switching vocoders?

No. The synthesizer and vocoder are decoupled components. The synthesizer outputs standard mel-spectrograms, so you can swap vocoders—including custom-trained ones—without retraining the Tacotron model or modifying the synthesis logic.

Which hyper-parameters must match between the synthesizer and custom vocoder?

The critical parameters defined in vocoder/hparams.py that must align include sample_rate, hop_length, num_mels, and mel_max_abs_value. Mismatches in these values will cause shape errors or audio pitch distortions during inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}