How to Diagnose and Fix Quality Issues in Synthesized Audio Output

You can diagnose and fix quality issues in synthesized audio output by systematically verifying the 256‑dimensional speaker encoder embedding, visualizing the Tacotron alignment matrix for diagonal continuity, and tuning Wave‑RNN hyper‑parameters such as tts_stop_threshold and bits to eliminate robotic artifacts, speaker mismatch, and distortion.

The Real‑Time‑Voice‑Cloning repository by CorentinJ implements a three‑stage neural pipeline: a speaker encoder extracts embeddings from reference audio, a Tacotron‑2 synthesizer generates mel‑spectrograms from text, and a Wave‑RNN vocoder renders the final waveform. When the output sounds like the wrong speaker, contains metallic ringing, or cuts off prematurely, the root cause usually resides in one of these stages. This guide provides a systematic workflow to isolate and resolve quality issues using diagnostics built into the source code.

Verify the Speaker Embedding in encoder/inference.py

Quality problems often begin with a corrupted or poorly normalized speaker embedding. In encoder/inference.py, the embed_utterance function computes a 256‑dimensional vector that conditions the synthesizer on the target voice.

Check for these specific failure modes:

• Speech sounds like the wrong speaker or is too generic. Calculate the embedding norm using np.linalg.norm(embed) immediately after calling embed_utterance. A norm far from 1.0 indicates the vector was not L2‑normalized correctly, causing the synthesizer to ignore speaker characteristics.
• Output contains NaN or infinite values. Run np.isnan(embed).any() or np.isinf(embed).any() to detect numerical instability, which typically stems from clipped or silent reference audio.

To fix embedding issues, preprocess the reference wav with encoder/audio.preprocess_wav, ensuring the input is a clean, ~3‑second clip sampled at 16 kHz. Enable the rescaling step by setting hparams.rescale to True and verify that hparams.rescaling_max is set to 0.99 in encoder/audio.py to prevent amplitude overflow.

Inspect Tacotron Alignment and Spectrograms

The synthesizer stage in synthesizer/inference.py constructs an alignment matrix between text characters and mel frames. A broken diagonal pattern in this matrix causes garbled speech or premature stopping.

Generate and visualize the alignment by passing return_alignments=True to synthesize_spectrograms:

from pathlib import Path
from synthesizer.inference import Synthesizer
from synthesizer.utils.plot import plot_alignment

synth = Synthesizer(Path("saved_models/synthesizer.pt"))
specs, align = synth.synthesize_spectrograms(
    ["Hello world"], embed, return_alignments=True
)

# Visualize the alignment matrix

plot_alignment(align[0], "alignment.png", title="Tacotron Alignment")

Interpret the visualization using these criteria:

• Diagonal is broken or jagged. Abrupt jumps indicate the decoder stopped too early. Increase tts_stop_threshold (default 0.5) in synthesizer/hparams.py to allow more timesteps before termination.
• Spectrogram is too dark or low‑energy. If mel values cluster near zero, confirm that hparams.rescale and hparams.rescaling_max (default 0.99) are active. Under‑amplified signals produce muffled output.
• Excessive silence at the end. Long trailing columns of zeros suggest the stop threshold is too high. Decrease tts_stop_threshold or retrain with silence‑trimmed data.

Diagnose the Wave‑RNN Vocoder

The vocoder stage in vocoder/inference.py converts mel frames to a waveform. Artefacts such as metallic ringing, high‑frequency noise, or clicking usually trace back to vocoder/hparams.py configuration.

Run the vocoder inference and inspect the output:

from vocoder.inference import load_model, infer_waveform
import soundfile as sf
from pathlib import Path

load_model(Path("saved_models/vocoder/wavernn.pt"))
wav = infer_waveform(mel, batched=False)  # mel from synthesizer

sf.write("output.wav", wav.cpu().numpy(), samplerate=22050)

Address these specific vocoder issues:

• Robotic or crunchy voice. Low bit‑depth causes quantization noise. Increase hp.bits from 8 or 9 to 16 in vocoder/hparams.py. Note that changing this requires retraining the vocoder.
• Muffled high frequencies. Verify that hp.sample_rate in the vocoder matches the synthesizer’s rate (typically 22050). A mismatch between 22050 and 24000 cuts off upper harmonics.
• Clicking at segment boundaries. Increase hp.voc_pad (default 2) to provide larger padding for upsampling layers, or retrain with larger padding values to reduce edge artefacts.
• High noise floor. Ensure you call infer_waveform with normalize=True so mel values scale correctly by hp.mel_max_abs_value.

End‑to‑End Validation and Hyper‑Parameter Tuning

After adjusting individual components, run a complete pipeline test using the command‑line demo:

python demo_cli.py \
    --enc-model saved_models/encoder.pt \
    --synth-model saved_models/synthesizer.pt \
    --vocoder-model saved_models/vocoder/wavernn.pt \
    --input-text "Testing the voice cloning pipeline." \
    --reference-audio samples/clean_reference.wav \
    --output output.wav

If abnormalities persist, adjust these critical hyper‑parameters and reload the models:

• synthesizer/hparams.py: tts_stop_threshold (0.3–0.7) controls when Tacotron stops emitting frames. Lower values reduce trailing silence; higher values prevent early cutoff.
• synthesizer/hparams.py: rescale / rescaling_max (True / 0.99) normalizes input amplitude for the encoder.
• vocoder/hparams.py: bits (8–16) sets Wave‑RNN output bit‑depth.
• vocoder/hparams.py: sample_rate (22050 or 24000) must match the synthesizer.
• vocoder/hparams.py: voc_pad (2–5) reduces boundary clicking during upsampling.

Changes to hparams.py take effect immediately upon reloading the model or restarting the inference script.

Summary

• Check embedding integrity using np.linalg.norm and np.isnan in encoder/inference.py to ensure the speaker vector is normalized and finite.
• Visualize Tacotron alignment with plot_alignment from synthesizer/utils/plot.py; adjust tts_stop_threshold if the diagonal is broken.
• Normalize input audio via encoder/audio.preprocess_wav with rescale=True to prevent low‑energy spectrograms.
• Match sample rates between synthesizer/hparams.py and vocoder/hparams.py (both should be 22050).
• Increase bit‑depth by setting hp.bits to 16 in vocoder/hparams.py to eliminate robotic artefacts.
• Validate end‑to‑end with demo_cli.py after each hyper‑parameter change to isolate residual quality issues.

Frequently Asked Questions

Why does my synthesized voice sound like the wrong speaker?

This occurs when the speaker embedding extracted by encoder.inference.embed_utterance has an L2 norm significantly different from 1.0 or contains NaN values due to noisy reference audio. Preprocess the reference clip using encoder.audio.preprocess_wav with rescale=True and ensure the input is a clean, single‑speaker wav file sampled at 16 kHz.

How do I fix robotic or metallic audio artifacts?

Robotic sound indicates low quantization bit‑depth in the vocoder. Open vocoder/hparams.py and increase hp.bits from the default 8 or 9 up to 16. This requires retraining the Wave‑RNN model on your dataset, but eliminates the crunchy, low‑bit artefacts characteristic of insufficient quantization.

What causes clicking noises at the end of generated audio clips?

Clicking stems from insufficient padding at upsampling layer boundaries in the vocoder. In vocoder/hparams.py, raise hp.voc_pad from its default of 2 to 4 or 5. Alternatively, verify that tts_stop_threshold in synthesizer/hparams.py is not set too low, which can cause the synthesizer to truncate the mel‑spectrogram abruptly.

How can I visualize the Tacotron alignment matrix to debug mispronunciations?

Import plot_alignment from synthesizer.utils.plot and call synth.synthesize_spectrograms with return_alignments=True. Save the resulting matrix using plot_alignment(align[0], "debug.png"). A healthy alignment shows a smooth diagonal line from top‑left to bottom‑right; gaps or jagged breaks indicate that the text‑to‑mel attention mechanism has failed to converge, usually requiring adjustment of tts_stop_threshold or additional training data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}