# How to Integrate a Custom-Trained Vocoder with the Real-Time-Voice-Cloning Synthesizer > Easily integrate a custom-trained vocoder with Real-Time-Voice-Cloning. Train WaveRNN, save checkpoint, and load it for seamless synthesis. Get high-quality custom voice cloning today. - Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning) - Tags: how-to-guide - Published: 2026-03-05 --- **You can integrate a custom-trained vocoder into the Real-Time-Voice-Cloning pipeline by training a WaveRNN model using the provided scripts, saving the checkpoint, and loading it via `vocoder.inference.load_model()` before calling `infer_waveform()` on mel-spectrograms generated by the synthesizer.** The CorentinJ/Real-Time-Voice-Cloning repository implements a modular voice cloning architecture where the synthesizer and vocoder communicate through a standardized mel-spectrogram interface. Because the synthesizer only outputs NumPy arrays representing mel-spectrograms, you can integrate a custom-trained vocoder without modifying the core synthesis code, provided your model respects the shared hyper-parameters. This separation allows you to swap the default WaveRNN with your own trained checkpoint to achieve custom audio characteristics or improved inference speed. ## Architecture Overview The repository organizes voice cloning into three sequential neural components: - **Encoder** – Extracts a 256-dimensional speaker embedding from reference audio. - **Synthesizer** – A Tacotron-based model that converts text and speaker embeddings into mel-spectrograms. The inference logic resides in [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py), specifically within the `Synthesizer.synthesize_spectrograms` method (lines 70‑84). - **Vocoder** – A WaveRNN model that transforms mel-spectrograms into raw waveforms. The entry points are `load_model` and `infer_waveform` in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 8‑46). The integration point between the synthesizer and vocoder is strictly the mel-spectrogram object (`np.ndarray`). Any vocoder architecture that accepts this tensor shape and the hyper-parameters defined in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) can replace the default implementation. ## Step-by-Step Integration Guide ### Step 1: Train Your Custom Vocoder Train your vocoder using the repository’s training scripts. You can use either the high-level wrapper [`vocoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder_train.py) or the underlying [`vocoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/train.py) module. The training process saves a checkpoint file named `.pt` containing the model weights and optimizer state. Place this checkpoint in your preferred directory, such as `models/custom/`, without modifying the repository structure. ### Step 2: Load the Custom Model at Runtime Before running inference, load your checkpoint by calling `vocoder.inference.load_model()` with the path to your `.pt` file. This function replaces the default WaveRNN instance stored in the internal `_model` variable, as implemented in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) (lines 8‑15). ```python from vocoder import inference as vocoder from pathlib import Path custom_vocoder_path = Path("models/custom/my_vocoder.pt") vocoder.load_model(custom_vocoder_path) ``` ### Step 3: Align Hyper-Parameters The vocoder relies on shared hyper-parameters defined in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py). If your training modified any settings—such as `voc_rnn_dims`, `bits`, or `sample_rate`—you must ensure these values match between the synthesizer and vocoder. Critical parameters that must align include: - `sample_rate` - `hop_length` - `num_mels` Edit [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) directly or import a modified configuration before loading your model to maintain compatibility. ### Step 4: Execute the Full Pipeline Generate audio by first creating mel-spectrograms with the synthesizer, then passing them to your custom vocoder: 1. Instantiate the synthesizer with a trained Tacotron checkpoint using [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py). 2. Generate mel-spectrograms via `synthesizer.synthesize_spectrograms(texts, speaker_embeddings)`. 3. Convert the spectrograms to waveforms using `vocoder.infer_waveform(mel_spec)`. ## Complete Implementation Example The following example demonstrates loading both the synthesizer and a custom vocoder, then generating audio from text: ```python # ------------------------------------------------- # 1️⃣ Load the Synthesizer (pre‑trained Tacotron) # ------------------------------------------------- from pathlib import Path from synthesizer.inference import Synthesizer synth_path = Path("saved_models/synthesizer.pt") synth = Synthesizer(synth_path) # ------------------------------------------------- # 2️⃣ Generate a mel‑spectrogram for some text # ------------------------------------------------- texts = ["Hello, this is a custom vocoder test."] import numpy as np # Use a zero‑vector speaker embedding for demonstration speaker_emb = np.zeros((256,)) # Replace with encoder output in production mel_specs = synth.synthesize_spectrograms(texts, speaker_emb)[0] # ------------------------------------------------- # 3️⃣ Load your custom‑trained vocoder # ------------------------------------------------- from vocoder import inference as vocoder custom_vocoder_path = Path("models/custom/my_vocoder.pt") vocoder.load_model(custom_vocoder_path) # ------------------------------------------------- # 4️⃣ Convert the mel‑spectrogram to waveform # ------------------------------------------------- wav = vocoder.infer_waveform(mel_specs) # ------------------------------------------------- # 5️⃣ Save or play the waveform # ------------------------------------------------- import soundfile as sf sf.write("output.wav", wav.cpu().numpy(), samplerate=22050) # sample_rate from hparams ``` Key implementation details from the source code: - `Synthesizer.synthesize_spectrograms` returns a list of mel arrays compatible with the vocoder input requirements. - `vocoder.load_model` hot-swaps the default WaveRNN with your custom checkpoint. - `vocoder.infer_waveform` automatically normalizes the mel-spectrogram using `hp.mel_max_abs_value` constants defined during training. ## Critical Files for Integration | File | Role | Key Components | |------|------|----------------| | [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) | Tacotron-based synthesizer | `Synthesizer.synthesize_spectrograms` (lines 70‑84) | | [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) | Vocoder inference wrapper | `load_model`, `infer_waveform` (lines 8‑46) | | [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) | Shared hyper-parameters | `sample_rate`, `hop_length`, `num_mels`, `mel_max_abs_value` | | [`vocoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/train.py) | Training script | Entry point for custom vocoder training | | [`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py) | Reference WaveRNN implementation | Default architecture used by the repository | ## Summary - The **mel-spectrogram** serves as the universal interface between the synthesizer and any custom vocoder. - Call **`vocoder.inference.load_model()`** to replace the default WaveRNN with your trained checkpoint. - Ensure **[`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py)** values match your training configuration to prevent audio distortion. - No modifications to [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) are required; the synthesizer operates independently of the vocoder implementation. ## Frequently Asked Questions ### Can I use a vocoder architecture other than WaveRNN? Yes. You can integrate any vocoder architecture—such as HiFi-GAN or Griffin-Lim—provided it accepts a mel-spectrogram with the shape `(num_mels, time)` and the sample rate specified in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py). You must wrap your model with a compatible `load_model` and `infer_waveform` interface similar to [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py). ### Where exactly does the synthesizer output the mel-spectrogram? According to the source code in [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py), the `Synthesizer.synthesize_spectrograms` method (lines 70‑84) processes input text and speaker embeddings through the Tacotron model and returns a Python list containing NumPy arrays, where each array represents a mel-spectrogram. ### Do I need to retrain the synthesizer when switching vocoders? No. The synthesizer and vocoder are decoupled components. The synthesizer outputs standard mel-spectrograms, so you can swap vocoders—including custom-trained ones—without retraining the Tacotron model or modifying the synthesis logic. ### Which hyper-parameters must match between the synthesizer and custom vocoder? The critical parameters defined in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) that must align include `sample_rate`, `hop_length`, `num_mels`, and `mel_max_abs_value`. Mismatches in these values will cause shape errors or audio pitch distortions during inference.