# Essential Preprocessing Steps Before Training the Synthesizer in Real-Time-Voice-Cloning

> Discover essential preprocessing steps for Real-Time-Voice-Cloning. Convert audio to mel-spectrograms and metadata through normalization, trimming, and mel computation.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: how-to-guide
- Published: 2026-03-05

---

**Before training the Tacotron-based synthesizer in the Real-Time-Voice-Cloning project, you must convert raw speech recordings into mel-spectrograms and structured metadata by executing dataset discovery, audio normalization, silence trimming, and mel computation via [`synthesizer_preprocess_audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer_preprocess_audio.py).**

The synthesizer in CorentinJ/Real-Time-Voice-Cloning implements a Tacotron-based architecture that requires strictly formatted input tensors. These preprocessing steps essential before training the synthesizer transform raw WAV, FLAC, or MP3 files into normalized mel-spectrograms and populate the [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) metadata file that [`synthesizer_train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer_train.py) consumes.

## Core Preprocessing Pipeline

The pipeline is orchestrated by `preprocess_dataset()` in [`synthesizer/preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/preprocess.py) and follows a deterministic sequence to ensure training stability and data consistency.

### Dataset Discovery and Directory Setup

The process begins by scanning the specified dataset root (e.g., `datasets_root/LibriSpeech/train-clean-100`) to identify all speaker directories. According to the source code in [`synthesizer/preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/preprocess.py) (lines 13-20), the function creates output subdirectories `mels/` and `audio/` under the target path and initializes (or appends to) the [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) metadata file that tracks every processed utterance.

### Audio Loading and Normalization

For each utterance, the pipeline loads the waveform using `librosa.load(str(wav_fpath), hparams.sample_rate)` at the sample rate defined in your hyperparameters. If `hparams.rescale` is enabled, the audio is normalized to a target amplitude via:

```python
wav = wav / np.abs(wav).max() * hparams.rescaling_max

```

The encoder's `preprocess_wav()` function then optionally trims leading and trailing silence (`trim_silence=True`) while skipping normalization to avoid double-processing, as implemented in the utterance processing logic of [`synthesizer/preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/preprocess.py).

### Text Extraction and Utterance Splitting

The system reads accompanying transcript files (`.txt` or [`.normalized.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/.normalized.txt)), cleaning quotes and whitespace to produce normalized text. For datasets providing alignment files (e.g., original LibriSpeech), the `split_on_silences()` function segments long recordings into sub-utterances based on silence detection. The pipeline discards any audio shorter than `hparams.utterance_min_duration` to ensure sufficient context for the model.

### Mel-Spectrogram Computation and Filtering

The critical transformation occurs in [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py), where `audio.melspectrogram(wav, hparams)` converts the processed waveform into a mel-spectrogram. The pipeline filters out utterances exceeding `hparams.max_mel_frames` to prevent out-of-memory errors during training. Valid mel arrays are saved as `.npy` files in the `mels/` directory, while the corresponding waveforms are stored in `audio/`.

### Metadata Recording and Speaker Embeddings

Each successful utterance generates a pipe-separated line in [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) following the format `[audio-file|mel-file|embed-file|audio-len|mel-frames|text]`, written via `metadata_file.write()` in [`synthesizer/preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/preprocess.py). For the full SV2TTS pipeline, you must subsequently run `create_embeddings()` (exposed via [`synthesizer_preprocess_embeds.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer_preprocess_embeds.py)) to generate speaker embedding files in `embeds/` using the pretrained encoder model.

## How to Execute the Preprocessing

Run the audio preprocessing from the repository root, specifying your dataset location and desired subfolders:

```bash
python synthesizer_preprocess_audio.py /path/to/datasets_root \
    -o /path/to/output/synthesizer \
    -n 8 \
    --no_alignments \
    --datasets_name LibriSpeech \
    --subfolders train-clean-100,train-clean-360

```

For datasets with alignment files (original LibriSpeech), omit `--no_alignments`:

```bash
python synthesizer_preprocess_audio.py /data/LibriSpeech \
    -o ./SV2TTS/synthesizer \
    -n 4 \
    --datasets_name LibriSpeech \
    --subfolders train-clean-100

```

After generating mel and audio files, create speaker embeddings:

```bash
python synthesizer_preprocess_embeds.py ./SV2TTS/synthesizer \
    -e ./pretrained_models/encoder.pt \
    -n 8

```

## Summary

- **Dataset Discovery**: `preprocess_dataset()` in [`synthesizer/preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/preprocess.py) scans speaker directories and initializes output folders (`mels/`, `audio/`) and the [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) metadata file.
- **Audio Normalization**: Load with `librosa.load()`, rescale via `hparams.rescaling_max`, and trim silence using `encoder.preprocess_wav()` from [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py).
- **Mel Generation**: Compute spectrograms with `audio.melspectrogram()` in [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py) and filter by `hparams.utterance_min_duration` and `hparams.max_mel_frames`.
- **Metadata Creation**: Write pipe-separated entries to [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) and generate speaker embeddings with `create_embeddings()` for the complete SV2TTS pipeline.

## Frequently Asked Questions

### What file format does the synthesizer training script expect?

The synthesizer expects precomputed mel-spectrograms stored as `.npy` arrays in the `mels/` directory and a [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) file containing pipe-separated metadata lines that map audio files to their corresponding mel files, embeddings, and transcript text. The training script reads these mappings to load batches efficiently during Tacotron training.

### Why is silence trimming applied before mel-spectrogram computation?

Silence trimming, performed by `encoder.preprocess_wav()` with `trim_silence=True`, removes non-speech segments that provide no linguistic information but consume model capacity and increase sequence length. This ensures the Tacotron model trains only on meaningful acoustic features, improving convergence speed and synthesis quality.

### Can I skip the embedding creation step if I only want to train the synthesizer?

While the synthesizer can technically train without embeddings if you modify the data loader, the standard SV2TTS architecture implemented in this repository requires speaker embeddings to condition the Tacotron model on voice identity. You must run [`synthesizer_preprocess_embeds.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer_preprocess_embeds.py) to populate the `embeds/` directory and update [`train.txt`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/train.txt) with valid embedding paths, otherwise the training script will raise file-not-found errors.

### What happens if an utterance is too short or too long?

The pipeline automatically filters utterances shorter than `hparams.utterance_min_duration` (typically 1.6 seconds) or exceeding `hparams.max_mel_frames` (usually 900 frames), returning `None` for those samples and excluding them from the metadata file. This maintains consistent batch sizes during training and prevents out-of-memory errors from excessively long sequences.