How to Train the Speaker Encoder with Custom Datasets in Real-Time Voice Cloning

Training the speaker encoder on custom datasets requires three phases: preprocessing audio into mel-spectrograms using encoder_preprocess.py, executing the GE2E training loop via encoder_train.py, and optionally adjusting hyper-parameters in encoder/params_model.py to match your data distribution.

The SV2TTS (Speaker Verification to Text-to-Speech) framework in CorentinJ/Real-Time-Voice-Cloning relies on a Generalized End-to-End (GE2E) speaker encoder to generate speaker embeddings. While the repository includes pretrained weights, training the speaker encoder with custom datasets allows you to adapt the model to specific acoustic domains or languages not covered by the original VoxCeleb and LibriSpeech training data.

Organizing Custom Data for the Encoder Pipeline

Before launching the training pipeline, organize your audio files under a single root directory. The encoder_preprocess.py script recursively discovers WAV files within this structure, regardless of subfolder organization, making it compatible with arbitrary dataset layouts.

If your dataset format is not one of the three built-in options (VoxCeleb1, VoxCeleb2, or LibriSpeech), extend the preprocess_func dictionary in encoder_preprocess.py to map your dataset identifier to a custom preprocessing function. This function should receive the dataset path and return a list of speaker directories containing utterance paths, following the pattern established by the existing librispeech_other, voxceleb1, and voxceleb2 entries.

Phase 1: Preprocessing Raw Audio into Mel-Spectrograms

The encoder consumes normalized mel-filterbank representations rather than raw waveforms. The preprocessing script converts your audio into 40-channel mel-spectrograms using parameters defined in encoder/params_data.py (25 ms FFT windows, 10 ms step sizes, 16 kHz sampling rate, WebRTC VAD silence trimming).

Execute encoder_preprocess.py with your dataset root and desired dataset identifiers:

python encoder_preprocess.py ./MyDatasets \
    -d my_custom_dataset1,my_custom_dataset2 \
    -o ./MyDatasets/SV2TTS/encoder \
    --skip_existing

Key arguments include:

• datasets_root: Path containing your raw audio files
• --datasets (-d): Comma-separated list of dataset identifiers matching keys in preprocess_func
• --out_dir (-o): Destination for .npy mel-spectrogram files (defaults to <datasets_root>/SV2TTS/encoder/)
• --skip_existing: Skips recomputation of existing files during interrupted runs
• --no_trim: Disables WebRTC Voice Activity Detection (VAD) silence trimming

The script writes normalized mel-spectrograms as .npy files to the specified output directory, creating the clean_data_root required for the training phase.

Phase 2: Training the Speaker Encoder

The encoder_train.py script orchestrates the GE2E training loop defined in encoder/train.py. It instantiates the SpeakerEncoder class from encoder/model.py, which implements an LSTM-based embedding network that produces 256-dimensional speaker vectors optimized by the GE2E loss function.

Launching the Training Script

Start training with a unique run identifier and point to your preprocessed data:

python encoder_train.py my_custom_encoder \
    ./MyDatasets/SV2TTS/encoder \
    -m saved_models \
    -v 10 -u 200 -s 500 -b 2000 \
    --visdom_server http://localhost

Critical parameters include:

• run_id: Experiment name determining the subdirectory under saved_models/
• clean_data_root: Path to preprocessed mel-spectrograms from Phase 1
• --models_dir (-m): Checkpoint storage location
• --vis_every (-v): Steps between Visdom visualization updates handled by encoder/visualizations.py
• --umap_every (-u): Steps between UMAP projection plots saved as umap_<step>.png
• --save_every (-s): Steps between model checkpoints
• --backup_every (-b): Steps between backup copies
• --force_restart (-f): Ignores existing checkpoints to start training from scratch

Understanding the GE2E Training Loop

The SpeakerVerificationDataset class in encoder/data_objects/speaker_verification_dataset.py groups utterances by speaker identity, enabling the GE2E loss to compute cosine similarities between intra-speaker and inter-speaker embedding clusters. The loss function, implemented in encoder/model.py via the similarity_matrix computation, optimizes the embedding space to maximize same-speaker coherence while minimizing cross-speaker overlap.

The training loop in encoder/train.py handles batching through do_gradient_ops, loss computation, checkpointing via save_every intervals, and optional Visdom visualization when a server is available.

Phase 3: Fine-Tuning Hyper-Parameters

Edit encoder/params_model.py before training to adjust model architecture and training dynamics for your custom data:

• learning_rate_init: Initial learning rate for the optimizer
• speakers_per_batch: Number of distinct speakers per training batch
• utterances_per_speaker: Number of utterances sampled per speaker for GE2E loss computation

These parameters directly affect GPU memory consumption and convergence characteristics when working with custom datasets that may differ from the original VoxCeleb distribution. The model architecture parameters (LSTM hidden sizes, embedding dimensions) also reside in this file.

Loading Trained Encoders for Inference

Upon completion, the final weights are saved to <models_dir>/<run_id>/encoder.pt. Load the trained encoder using the inference API:

from encoder import inference

encoder = inference.load_pretrained_model('saved_models/my_custom_encoder/encoder.pt')
wav = inference.load_audio('new_voice.wav')
embedding = inference.embed_utterance(encoder, wav)
print(f"Speaker embedding shape: {embedding.shape}")

The resulting 256-dimensional vector can condition the synthesizer stage of the SV2TTS pipeline or be used for standalone speaker verification tasks.

Summary

• Preprocessing: Convert raw audio to mel-spectrograms using encoder_preprocess.py, which applies VAD-based silence trimming defined in encoder/params_data.py and stores .npy files in SV2TTS/encoder/
• Training: Execute encoder_train.py with your run_id and clean_data_root to launch the GE2E loop managed by encoder/train.py, utilizing the SpeakerEncoder LSTM architecture from encoder/model.py
• Configuration: Modify encoder/params_model.py to adjust batch composition (speakers_per_batch, utterances_per_speaker) and learning rates for custom data distributions
• Dataset Integration: Add custom dataset support by extending the preprocess_func dictionary in encoder_preprocess.py to handle non-standard directory structures
• Output: Trained models are saved to <models_dir>/<run_id>/encoder.pt and loaded via encoder/inference.py for embedding extraction

Frequently Asked Questions

Can I train the encoder on datasets other than VoxCeleb or LibriSpeech?

Yes. Add a custom entry to the preprocess_func dictionary in encoder_preprocess.py that maps your dataset identifier to a preprocessing function returning speaker-labeled utterance paths. The script will then handle your data exactly like the built-in datasets, outputting compatible mel-spectrograms to SV2TTS/encoder/ for consumption by the training pipeline.

Where are the audio preprocessing parameters defined?

The mel-filterbank configuration (40 channels, 25 ms windows, 10 ms steps, 16 kHz sampling) is defined in encoder/params_data.py. This file also controls WebRTC VAD aggressiveness for silence trimming during preprocessing. Modify these values if your custom data requires different spectral resolution or sampling rates.

How do I adjust the training batch composition for my hardware?

Edit speakers_per_batch and utterances_per_speaker in encoder/params_model.py before launching encoder_train.py. These parameters control the GE2E loss computation matrix size and directly impact GPU memory requirements; reduce them if encountering CUDA out-of-memory errors during training.

How do I resume training from a checkpoint versus starting fresh?

Remove the --force_restart (-f) flag when calling encoder_train.py to automatically resume from the latest checkpoint in saved_models/<run_id>/. To restart training from random initialization, include the -f flag, which instructs the training loop in encoder/train.py to ignore existing model weights and optimizer states.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}