# Common Failure Modes in Voice Cloning: Diagnostic Steps for Real-Time Voice Cloning

> Diagnose common voice cloning failures like missing checkpoints or audio issues. Learn essential troubleshooting steps for real-time voice cloning to ensure smooth operation and high-quality results.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: best-practices
- Published: 2026-03-05

---

**The most common voice cloning failures stem from missing model checkpoints, incompatible audio formats (non-16kHz/mono), dimensional mismatches between mel-spectrograms and waveforms, and GPU batch-size misconfigurations, each throwing specific exceptions in the Encoder, Synthesizer, or Vocoder stages that can be diagnosed by verifying file paths, audio preprocessing, and tensor shapes.**

Real-Time Voice Cloning (SV2TTS) by CorentinJ implements a three-stage pipeline that frequently fails during model loading, audio preprocessing, or tensor alignment. Understanding these voice cloning failure modes requires tracing exceptions through the **Encoder**, **Synthesizer**, and **Vocoder** components to identify whether the issue originates from data preparation, checkpoint availability, or hardware constraints.

## Encoder Stage Failures and Diagnostic Steps

The **Encoder** extracts fixed-dimensional speaker embeddings from short audio samples. Failures here typically involve missing checkpoints or malformed input audio.

### Missing Model Checkpoints

In [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) at line 51, the code raises `Exception("Model was not loaded…")` if inference is attempted before loading weights. This occurs when `encoder/saved_models/` lacks `.pth` files or the path is misconfigured.

Implement safe model loading with explicit diagnostics to catch these errors before inference:

```python
def load_all():
    from utils import default_models
    import pathlib, sys

    try:
        enc = default_models.get_pretrained_encoder()
    except Exception as e:
        sys.exit(f"❌ Encoder load error: {e}")

    try:
        syn = default_models.get_pretrained_synthesizer()
    except Exception as e:
        sys.exit(f"❌ Synthesizer load error: {e}")

    try:
        voc = default_models.get_pretrained_vocoder()
    except Exception as e:
        sys.exit(f"❌ Vocoder load error: {e}")

    print("✅ All models loaded successfully")
    return enc, syn, voc

```

**Diagnostic steps:**
- Verify `encoder/saved_models/` contains `*.pth` checkpoints
- Call `load_model()` before any embedding extraction
- Use `utils/default_models.get_pretrained_encoder()` to auto-download official weights

### Unsupported Audio Formats

[`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) asserts dataset existence at line 58 and strictly expects 16kHz mono PCM. Non-compliant files trigger silent failures or preprocessing errors during speaker embedding extraction.

**Diagnostic steps:**
1. Run `ffprobe <file>` or `ffmpeg -i <file>` to confirm 16kHz sample rate and 1-channel layout
2. Re-encode non-compliant files using: `ffmpeg -i input.wav -ar 16000 -ac 1 output.wav`

### Empty or Silent Audio Clips

[`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py) raises `ValueError("Both increase only and decrease only are set")` at line 113 when processing all-zero signals or extremely low-amplitude audio.

Validate audio preprocessing before encoding to catch silent inputs:

```python
import soundfile as sf
import numpy as np

def check_wav(path):
    wav, sr = sf.read(path)
    if sr != 16000:
        raise ValueError(f"Sample rate {sr} Hz, expected 16kHz")
    if wav.ndim != 1:
        raise ValueError("Audio must be mono")
    if np.max(np.abs(wav)) > 1.0:
        raise ValueError("Audio amplitude exceeds [-1, 1]")
    if np.mean(np.abs(wav)) < 1e-4:
        raise ValueError("Audio appears silent")
    print("✅ WAV file is valid")

```

**Diagnostic steps:**
- Calculate RMS energy: `np.sqrt(np.mean(wav**2))`
- Ensure recordings contain at least 2 seconds of speech with amplitude > 1e-4

## Synthesizer Stage Failures and Diagnostic Steps

The **Synthesizer** generates mel-spectrograms from text and speaker embeddings. Failures involve dimensional mismatches, training configuration errors, or missing trained models.

### Mel-Spectrogram Dimension Mismatches

[`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) expects mel frames with exactly `hparams.num_mels` columns. Mismatches trigger assertions in [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py) at line 169: `assert hparams.fmax <= hparams.sample_rate // 2`.

**Diagnostic steps:**
- After generation, verify `print(mel.shape)` matches `hparams.num_mels`
- Confirm `fmax` does not exceed the Nyquist frequency (`sample_rate // 2`)

### GPU Batch-Size Configuration Errors

[`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py) enforces multi-GPU divisibility at line 60: `ValueError("`batch_size` must be evenly divisible by n_gpus!")`.

**Diagnostic steps:**
- Set `batch_size` to a multiple of your GPU count (e.g., 8, 16, or 32 for 4 GPUs)
- Force CPU inference with `--extra cpu` to bypass GPU constraints entirely

### Missing Trained Models

[`toolbox/ui.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/toolbox/ui.py) raises `Exception("No synthesizer models found in %s")` at line 344 when the GUI cannot locate synthesizer checkpoints.

**Diagnostic steps:**
- Verify `synthesizer/saved_models/` contains `*.pth` files
- Call `utils/default_models.get_pretrained_synthesizer()` to download official weights

## Vocoder Stage Failures and Diagnostic Steps

The **Vocoder** (WaveRNN) converts mel-spectrograms to raw waveforms. Failures typically involve length misalignment, undefined model modes, or uninitialized model memory.

### Mel-Length vs Waveform Alignment Errors

[`vocoder/vocoder_dataset.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/vocoder_dataset.py) validates strict alignment at lines 39 and 41:
- `assert len(wav) >= mel.shape[1] * hp.hop_length`
- `assert len(wav) % hp.hop_length == 0`

**Diagnostic steps:**
- Check `wav.shape` versus `mel.shape` before feeding to [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py)
- Pad or truncate the mel so that waveform length is an integer multiple of `hop_length` (default 200 samples)

### WaveRNN Model Mode Errors

[`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py) raises `RuntimeError("Unknown model mode value - ", self.mode)` at line 230 when loading checkpoints with incompatible mode configurations.

**Diagnostic steps:**
- Ensure checkpoints specify `model_mode='rnn'` for WaveRNN
- Inspect the checkpoint dictionary: `checkpoint['model_mode']`

### Unloaded Model Memory Errors

[`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) raises `Exception("Please load Wave‑RNN in memory before using it")` at line 58 if `generate_waveform` is called before initialization.

**Diagnostic steps:**
- Explicitly call `vocoder_inf.load_model()` before inference
- Verify GPU memory availability (approximately 2GB VRAM for the default model)

## End-to-End Diagnostic Workflow

Run this sequential validation script to isolate failures across all three stages:

```python
from encoder import inference as encoder_inf
from synthesizer.inference import Synthesizer
from vocoder import inference as vocoder_inf
from utils import default_models

# Stage 1: Load pretrained models – catches missing-file errors

enc = default_models.get_pretrained_encoder()
syn = default_models.get_pretrained_synthesizer()
voc = default_models.get_pretrained_vocoder()

# Stage 2: Encode a short wav (must be 16kHz mono)

embed, _ = encoder_inf.embed_utterance("my_voice.wav")

# Stage 3: Generate mel – verify dimensions

mel = syn.synthesize_spectrogram("Hello world!", embed)
assert mel.shape[0] == syn.hparams.num_mels, "Unexpected mel channel count"

# Stage 4: Vocoder – check hop-length alignment

wav = voc.infer_waveform(mel)
assert len(wav) % voc.hparams.hop_length == 0, "Wave-length not aligned"

```

## Summary

- **Missing checkpoints** throw explicit exceptions in [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py), [`toolbox/ui.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/toolbox/ui.py), and [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py); resolve using [`utils/default_models.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/utils/default_models.py) helpers.
- **Audio format errors** originate in [`encoder_preprocess.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder_preprocess.py) and [`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py); enforce 16kHz mono PCM with minimum 2-second duration.
- **Dimensional mismatches** surface in [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py) and [`vocoder/vocoder_dataset.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/vocoder_dataset.py); verify `num_mels` and `hop_length` alignment before inference.
- **Hardware errors** appear in [`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py) as batch-size divisibility failures; adjust batch size to GPU count or use CPU fallback.

## Frequently Asked Questions

### Why does the voice cloning encoder raise "Model was not loaded" even when files exist?

This error in [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) line 51 occurs when `load_model()` was never called or the path points to a corrupted checkpoint. Ensure you explicitly call the loader and verify the `.pth` file is complete (not partially downloaded). Use `default_models.get_pretrained_encoder()` to bypass path configuration errors.

### How do I fix the "batch_size must be evenly divisible by n_gpus" error in the synthesizer?

[`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py) at line 60 requires the batch size to divide evenly across all GPUs. Either set the batch size to a multiple of your GPU count (e.g., 4, 8, or 12 for 2 GPUs) or force single-device training by passing `--extra cpu` to run on CPU only.

### What causes "Unknown model mode value" in the vocoder?

[`vocoder/models/fatchord_version.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/models/fatchord_version.py) raises this at line 230 when the checkpoint's `model_mode` key does not match the expected WaveRNN configuration. Ensure you load the official pretrained vocoder via `default_models.get_pretrained_vocoder()`, which sets `model_mode='rnn'` correctly.

### How do I diagnose silent or empty audio failures in voice cloning?

[`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py) line 113 throws `ValueError` when processing all-zero signals. Calculate the RMS energy of your input waveform before encoding; if `np.sqrt(np.mean(wav**2))` is below 1e-4, the clip is effectively silent. Re-record with at least 2 seconds of clear speech at 16kHz mono.