Essential Preprocessing Steps Before Training the Synthesizer in Real-Time-Voice-Cloning

Before training the Tacotron-based synthesizer in the Real-Time-Voice-Cloning project, you must convert raw speech recordings into mel-spectrograms and structured metadata by executing dataset discovery, audio normalization, silence trimming, and mel computation via synthesizer_preprocess_audio.py.

The synthesizer in CorentinJ/Real-Time-Voice-Cloning implements a Tacotron-based architecture that requires strictly formatted input tensors. These preprocessing steps essential before training the synthesizer transform raw WAV, FLAC, or MP3 files into normalized mel-spectrograms and populate the train.txt metadata file that synthesizer_train.py consumes.

Core Preprocessing Pipeline

The pipeline is orchestrated by preprocess_dataset() in synthesizer/preprocess.py and follows a deterministic sequence to ensure training stability and data consistency.

Dataset Discovery and Directory Setup

The process begins by scanning the specified dataset root (e.g., datasets_root/LibriSpeech/train-clean-100) to identify all speaker directories. According to the source code in synthesizer/preprocess.py (lines 13-20), the function creates output subdirectories mels/ and audio/ under the target path and initializes (or appends to) the train.txt metadata file that tracks every processed utterance.

Audio Loading and Normalization

For each utterance, the pipeline loads the waveform using librosa.load(str(wav_fpath), hparams.sample_rate) at the sample rate defined in your hyperparameters. If hparams.rescale is enabled, the audio is normalized to a target amplitude via:

wav = wav / np.abs(wav).max() * hparams.rescaling_max

The encoder's preprocess_wav() function then optionally trims leading and trailing silence (trim_silence=True) while skipping normalization to avoid double-processing, as implemented in the utterance processing logic of synthesizer/preprocess.py.

Text Extraction and Utterance Splitting

The system reads accompanying transcript files (.txt or .normalized.txt), cleaning quotes and whitespace to produce normalized text. For datasets providing alignment files (e.g., original LibriSpeech), the split_on_silences() function segments long recordings into sub-utterances based on silence detection. The pipeline discards any audio shorter than hparams.utterance_min_duration to ensure sufficient context for the model.

Mel-Spectrogram Computation and Filtering

The critical transformation occurs in synthesizer/audio.py, where audio.melspectrogram(wav, hparams) converts the processed waveform into a mel-spectrogram. The pipeline filters out utterances exceeding hparams.max_mel_frames to prevent out-of-memory errors during training. Valid mel arrays are saved as .npy files in the mels/ directory, while the corresponding waveforms are stored in audio/.

Metadata Recording and Speaker Embeddings

Each successful utterance generates a pipe-separated line in train.txt following the format [audio-file|mel-file|embed-file|audio-len|mel-frames|text], written via metadata_file.write() in synthesizer/preprocess.py. For the full SV2TTS pipeline, you must subsequently run create_embeddings() (exposed via synthesizer_preprocess_embeds.py) to generate speaker embedding files in embeds/ using the pretrained encoder model.

How to Execute the Preprocessing

Run the audio preprocessing from the repository root, specifying your dataset location and desired subfolders:

python synthesizer_preprocess_audio.py /path/to/datasets_root \
    -o /path/to/output/synthesizer \
    -n 8 \
    --no_alignments \
    --datasets_name LibriSpeech \
    --subfolders train-clean-100,train-clean-360

For datasets with alignment files (original LibriSpeech), omit --no_alignments:

python synthesizer_preprocess_audio.py /data/LibriSpeech \
    -o ./SV2TTS/synthesizer \
    -n 4 \
    --datasets_name LibriSpeech \
    --subfolders train-clean-100

After generating mel and audio files, create speaker embeddings:

python synthesizer_preprocess_embeds.py ./SV2TTS/synthesizer \
    -e ./pretrained_models/encoder.pt \
    -n 8

Summary

• Dataset Discovery: preprocess_dataset() in synthesizer/preprocess.py scans speaker directories and initializes output folders (mels/, audio/) and the train.txt metadata file.
• Audio Normalization: Load with librosa.load(), rescale via hparams.rescaling_max, and trim silence using encoder.preprocess_wav() from encoder/inference.py.
• Mel Generation: Compute spectrograms with audio.melspectrogram() in synthesizer/audio.py and filter by hparams.utterance_min_duration and hparams.max_mel_frames.
• Metadata Creation: Write pipe-separated entries to train.txt and generate speaker embeddings with create_embeddings() for the complete SV2TTS pipeline.

Frequently Asked Questions

What file format does the synthesizer training script expect?

The synthesizer expects precomputed mel-spectrograms stored as .npy arrays in the mels/ directory and a train.txt file containing pipe-separated metadata lines that map audio files to their corresponding mel files, embeddings, and transcript text. The training script reads these mappings to load batches efficiently during Tacotron training.

Why is silence trimming applied before mel-spectrogram computation?

Silence trimming, performed by encoder.preprocess_wav() with trim_silence=True, removes non-speech segments that provide no linguistic information but consume model capacity and increase sequence length. This ensures the Tacotron model trains only on meaningful acoustic features, improving convergence speed and synthesis quality.

Can I skip the embedding creation step if I only want to train the synthesizer?

While the synthesizer can technically train without embeddings if you modify the data loader, the standard SV2TTS architecture implemented in this repository requires speaker embeddings to condition the Tacotron model on voice identity. You must run synthesizer_preprocess_embeds.py to populate the embeds/ directory and update train.txt with valid embedding paths, otherwise the training script will raise file-not-found errors.

What happens if an utterance is too short or too long?

The pipeline automatically filters utterances shorter than hparams.utterance_min_duration (typically 1.6 seconds) or exceeding hparams.max_mel_frames (usually 900 frames), returning None for those samples and excluding them from the metadata file. This maintains consistent batch sizes during training and prevents out-of-memory errors from excessively long sequences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}