# How to Train the Speaker Encoder with Custom Datasets in Real-Time Voice Cloning

> Learn to train the speaker encoder with custom datasets for real-time voice cloning. Follow three key phases: preprocess audio, run GE2E training, and tune parameters. Get started now.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Training the speaker encoder on custom datasets requires three phases: preprocessing audio into mel-spectrograms using [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py), executing the GE2E training loop via [`encoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_train.py), and optionally adjusting hyper-parameters in [`encoder/params_model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_model.py) to match your data distribution.**

The SV2TTS (Speaker Verification to Text-to-Speech) framework in CorentinJ/Real-Time-Voice-Cloning relies on a Generalized End-to-End (GE2E) speaker encoder to generate speaker embeddings. While the repository includes pretrained weights, training the speaker encoder with custom datasets allows you to adapt the model to specific acoustic domains or languages not covered by the original VoxCeleb and LibriSpeech training data.

## Organizing Custom Data for the Encoder Pipeline

Before launching the training pipeline, organize your audio files under a single root directory. The [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) script recursively discovers WAV files within this structure, regardless of subfolder organization, making it compatible with arbitrary dataset layouts.

If your dataset format is not one of the three built-in options (VoxCeleb1, VoxCeleb2, or LibriSpeech), extend the `preprocess_func` dictionary in [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) to map your dataset identifier to a custom preprocessing function. This function should receive the dataset path and return a list of speaker directories containing utterance paths, following the pattern established by the existing `librispeech_other`, `voxceleb1`, and `voxceleb2` entries.

## Phase 1: Preprocessing Raw Audio into Mel-Spectrograms

The encoder consumes normalized **mel-filterbank representations** rather than raw waveforms. The preprocessing script converts your audio into 40-channel mel-spectrograms using parameters defined in [`encoder/params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_data.py) (25 ms FFT windows, 10 ms step sizes, 16 kHz sampling rate, WebRTC VAD silence trimming).

Execute [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) with your dataset root and desired dataset identifiers:

```bash
python encoder_preprocess.py ./MyDatasets \
    -d my_custom_dataset1,my_custom_dataset2 \
    -o ./MyDatasets/SV2TTS/encoder \
    --skip_existing

```

Key arguments include:

- `datasets_root`: Path containing your raw audio files
- `--datasets` (`-d`): Comma-separated list of dataset identifiers matching keys in `preprocess_func`
- `--out_dir` (`-o`): Destination for `.npy` mel-spectrogram files (defaults to `<datasets_root>/SV2TTS/encoder/`)
- `--skip_existing`: Skips recomputation of existing files during interrupted runs
- `--no_trim`: Disables WebRTC Voice Activity Detection (VAD) silence trimming

The script writes normalized mel-spectrograms as `.npy` files to the specified output directory, creating the `clean_data_root` required for the training phase.

## Phase 2: Training the Speaker Encoder

The [`encoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_train.py) script orchestrates the GE2E training loop defined in [`encoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/train.py). It instantiates the `SpeakerEncoder` class from [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py), which implements an LSTM-based embedding network that produces 256-dimensional speaker vectors optimized by the GE2E loss function.

### Launching the Training Script

Start training with a unique run identifier and point to your preprocessed data:

```bash
python encoder_train.py my_custom_encoder \
    ./MyDatasets/SV2TTS/encoder \
    -m saved_models \
    -v 10 -u 200 -s 500 -b 2000 \
    --visdom_server http://localhost

```

Critical parameters include:

- `run_id`: Experiment name determining the subdirectory under `saved_models/`
- `clean_data_root`: Path to preprocessed mel-spectrograms from Phase 1
- `--models_dir` (`-m`): Checkpoint storage location
- `--vis_every` (`-v`): Steps between Visdom visualization updates handled by [`encoder/visualizations.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/visualizations.py)
- `--umap_every` (`-u`): Steps between UMAP projection plots saved as `umap_<step>.png`
- `--save_every` (`-s`): Steps between model checkpoints
- `--backup_every` (`-b`): Steps between backup copies
- `--force_restart` (`-f`): Ignores existing checkpoints to start training from scratch

### Understanding the GE2E Training Loop

The `SpeakerVerificationDataset` class in [`encoder/data_objects/speaker_verification_dataset.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/data_objects/speaker_verification_dataset.py) groups utterances by speaker identity, enabling the GE2E loss to compute cosine similarities between intra-speaker and inter-speaker embedding clusters. The loss function, implemented in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py) via the `similarity_matrix` computation, optimizes the embedding space to maximize same-speaker coherence while minimizing cross-speaker overlap.

The training loop in [`encoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/train.py) handles batching through `do_gradient_ops`, loss computation, checkpointing via `save_every` intervals, and optional Visdom visualization when a server is available.

## Phase 3: Fine-Tuning Hyper-Parameters

Edit [`encoder/params_model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_model.py) before training to adjust model architecture and training dynamics for your custom data:

- `learning_rate_init`: Initial learning rate for the optimizer
- `speakers_per_batch`: Number of distinct speakers per training batch
- `utterances_per_speaker`: Number of utterances sampled per speaker for GE2E loss computation

These parameters directly affect GPU memory consumption and convergence characteristics when working with custom datasets that may differ from the original VoxCeleb distribution. The model architecture parameters (LSTM hidden sizes, embedding dimensions) also reside in this file.

## Loading Trained Encoders for Inference

Upon completion, the final weights are saved to `<models_dir>/<run_id>/encoder.pt`. Load the trained encoder using the inference API:

```python
from encoder import inference

encoder = inference.load_pretrained_model('saved_models/my_custom_encoder/encoder.pt')
wav = inference.load_audio('new_voice.wav')
embedding = inference.embed_utterance(encoder, wav)
print(f"Speaker embedding shape: {embedding.shape}")

```

The resulting 256-dimensional vector can condition the synthesizer stage of the SV2TTS pipeline or be used for standalone speaker verification tasks.

## Summary

- **Preprocessing**: Convert raw audio to mel-spectrograms using [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py), which applies VAD-based silence trimming defined in [`encoder/params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_data.py) and stores `.npy` files in `SV2TTS/encoder/`
- **Training**: Execute [`encoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_train.py) with your `run_id` and `clean_data_root` to launch the GE2E loop managed by [`encoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/train.py), utilizing the `SpeakerEncoder` LSTM architecture from [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py)
- **Configuration**: Modify [`encoder/params_model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_model.py) to adjust batch composition (`speakers_per_batch`, `utterances_per_speaker`) and learning rates for custom data distributions
- **Dataset Integration**: Add custom dataset support by extending the `preprocess_func` dictionary in [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) to handle non-standard directory structures
- **Output**: Trained models are saved to `<models_dir>/<run_id>/encoder.pt` and loaded via [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) for embedding extraction

## Frequently Asked Questions

### Can I train the encoder on datasets other than VoxCeleb or LibriSpeech?

Yes. Add a custom entry to the `preprocess_func` dictionary in [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) that maps your dataset identifier to a preprocessing function returning speaker-labeled utterance paths. The script will then handle your data exactly like the built-in datasets, outputting compatible mel-spectrograms to `SV2TTS/encoder/` for consumption by the training pipeline.

### Where are the audio preprocessing parameters defined?

The mel-filterbank configuration (40 channels, 25 ms windows, 10 ms steps, 16 kHz sampling) is defined in [`encoder/params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_data.py). This file also controls WebRTC VAD aggressiveness for silence trimming during preprocessing. Modify these values if your custom data requires different spectral resolution or sampling rates.

### How do I adjust the training batch composition for my hardware?

Edit `speakers_per_batch` and `utterances_per_speaker` in [`encoder/params_model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_model.py) before launching [`encoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_train.py). These parameters control the GE2E loss computation matrix size and directly impact GPU memory requirements; reduce them if encountering CUDA out-of-memory errors during training.

### How do I resume training from a checkpoint versus starting fresh?

Remove the `--force_restart` (`-f`) flag when calling [`encoder_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_train.py) to automatically resume from the latest checkpoint in `saved_models/<run_id>/`. To restart training from random initialization, include the `-f` flag, which instructs the training loop in [`encoder/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/train.py) to ignore existing model weights and optimizer states.