# How to Diagnose and Fix Quality Issues in Synthesized Audio Output

> Fix synthesized audio quality issues by checking speaker embeddings, visualizing Tacotron alignment, and tuning WaveRNN parameters for natural voice output.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: how-to-guide
- Published: 2026-03-05

---

**You can diagnose and fix quality issues in synthesized audio output by systematically verifying the 256‑dimensional speaker encoder embedding, visualizing the Tacotron alignment matrix for diagonal continuity, and tuning Wave‑RNN hyper‑parameters such as `tts_stop_threshold` and `bits` to eliminate robotic artifacts, speaker mismatch, and distortion.**

The Real‑Time‑Voice‑Cloning repository by CorentinJ implements a three‑stage neural pipeline: a speaker encoder extracts embeddings from reference audio, a Tacotron‑2 synthesizer generates mel‑spectrograms from text, and a Wave‑RNN vocoder renders the final waveform. When the output sounds like the wrong speaker, contains metallic ringing, or cuts off prematurely, the root cause usually resides in one of these stages. This guide provides a systematic workflow to isolate and resolve quality issues using diagnostics built into the source code.

## Verify the Speaker Embedding in encoder/inference.py

Quality problems often begin with a corrupted or poorly normalized speaker embedding. In [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py), the `embed_utterance` function computes a 256‑dimensional vector that conditions the synthesizer on the target voice.

Check for these specific failure modes:

- **Speech sounds like the wrong speaker or is too generic.** Calculate the embedding norm using `np.linalg.norm(embed)` immediately after calling `embed_utterance`. A norm far from 1.0 indicates the vector was not L2‑normalized correctly, causing the synthesizer to ignore speaker characteristics.
- **Output contains NaN or infinite values.** Run `np.isnan(embed).any()` or `np.isinf(embed).any()` to detect numerical instability, which typically stems from clipped or silent reference audio.

To fix embedding issues, preprocess the reference wav with `encoder/audio.preprocess_wav`, ensuring the input is a clean, ~3‑second clip sampled at 16 kHz. Enable the rescaling step by setting `hparams.rescale` to `True` and verify that `hparams.rescaling_max` is set to `0.99` in [`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py) to prevent amplitude overflow.

## Inspect Tacotron Alignment and Spectrograms

The synthesizer stage in [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) constructs an alignment matrix between text characters and mel frames. A broken diagonal pattern in this matrix causes garbled speech or premature stopping.

Generate and visualize the alignment by passing `return_alignments=True` to `synthesize_spectrograms`:

```python
from pathlib import Path
from synthesizer.inference import Synthesizer
from synthesizer.utils.plot import plot_alignment

synth = Synthesizer(Path("saved_models/synthesizer.pt"))
specs, align = synth.synthesize_spectrograms(
    ["Hello world"], embed, return_alignments=True
)

# Visualize the alignment matrix

plot_alignment(align[0], "alignment.png", title="Tacotron Alignment")

```

Interpret the visualization using these criteria:

- **Diagonal is broken or jagged.** Abrupt jumps indicate the decoder stopped too early. Increase `tts_stop_threshold` (default `0.5`) in [`synthesizer/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/hparams.py) to allow more timesteps before termination.
- **Spectrogram is too dark or low‑energy.** If mel values cluster near zero, confirm that `hparams.rescale` and `hparams.rescaling_max` (default `0.99`) are active. Under‑amplified signals produce muffled output.
- **Excessive silence at the end.** Long trailing columns of zeros suggest the stop threshold is too high. Decrease `tts_stop_threshold` or retrain with silence‑trimmed data.

## Diagnose the Wave‑RNN Vocoder

The vocoder stage in [`vocoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/inference.py) converts mel frames to a waveform. Artefacts such as metallic ringing, high‑frequency noise, or clicking usually trace back to [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) configuration.

Run the vocoder inference and inspect the output:

```python
from vocoder.inference import load_model, infer_waveform
import soundfile as sf
from pathlib import Path

load_model(Path("saved_models/vocoder/wavernn.pt"))
wav = infer_waveform(mel, batched=False)  # mel from synthesizer

sf.write("output.wav", wav.cpu().numpy(), samplerate=22050)

```

Address these specific vocoder issues:

- **Robotic or crunchy voice.** Low bit‑depth causes quantization noise. Increase `hp.bits` from 8 or 9 to 16 in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py). Note that changing this requires retraining the vocoder.
- **Muffled high frequencies.** Verify that `hp.sample_rate` in the vocoder matches the synthesizer’s rate (typically `22050`). A mismatch between `22050` and `24000` cuts off upper harmonics.
- **Clicking at segment boundaries.** Increase `hp.voc_pad` (default `2`) to provide larger padding for upsampling layers, or retrain with larger padding values to reduce edge artefacts.
- **High noise floor.** Ensure you call `infer_waveform` with `normalize=True` so mel values scale correctly by `hp.mel_max_abs_value`.

## End‑to‑End Validation and Hyper‑Parameter Tuning

After adjusting individual components, run a complete pipeline test using the command‑line demo:

```bash
python demo_cli.py \
    --enc-model saved_models/encoder.pt \
    --synth-model saved_models/synthesizer.pt \
    --vocoder-model saved_models/vocoder/wavernn.pt \
    --input-text "Testing the voice cloning pipeline." \
    --reference-audio samples/clean_reference.wav \
    --output output.wav

```

If abnormalities persist, adjust these critical hyper‑parameters and reload the models:

- **[`synthesizer/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/hparams.py)**: `tts_stop_threshold` (0.3–0.7) controls when Tacotron stops emitting frames. Lower values reduce trailing silence; higher values prevent early cutoff.
- **[`synthesizer/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/hparams.py)**: `rescale` / `rescaling_max` (`True` / `0.99`) normalizes input amplitude for the encoder.
- **[`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py)**: `bits` (8–16) sets Wave‑RNN output bit‑depth.
- **[`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py)**: `sample_rate` (22050 or 24000) must match the synthesizer.
- **[`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py)**: `voc_pad` (2–5) reduces boundary clicking during upsampling.

Changes to [`hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/hparams.py) take effect immediately upon reloading the model or restarting the inference script.

## Summary

- **Check embedding integrity** using `np.linalg.norm` and `np.isnan` in [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) to ensure the speaker vector is normalized and finite.
- **Visualize Tacotron alignment** with `plot_alignment` from [`synthesizer/utils/plot.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/utils/plot.py); adjust `tts_stop_threshold` if the diagonal is broken.
- **Normalize input audio** via `encoder/audio.preprocess_wav` with `rescale=True` to prevent low‑energy spectrograms.
- **Match sample rates** between [`synthesizer/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/hparams.py) and [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) (both should be `22050`).
- **Increase bit‑depth** by setting `hp.bits` to 16 in [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) to eliminate robotic artefacts.
- **Validate end‑to‑end** with [`demo_cli.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_cli.py) after each hyper‑parameter change to isolate residual quality issues.

## Frequently Asked Questions

### Why does my synthesized voice sound like the wrong speaker?

This occurs when the speaker embedding extracted by `encoder.inference.embed_utterance` has an L2 norm significantly different from 1.0 or contains NaN values due to noisy reference audio. Preprocess the reference clip using `encoder.audio.preprocess_wav` with `rescale=True` and ensure the input is a clean, single‑speaker wav file sampled at 16 kHz.

### How do I fix robotic or metallic audio artifacts?

Robotic sound indicates low quantization bit‑depth in the vocoder. Open [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py) and increase `hp.bits` from the default 8 or 9 up to 16. This requires retraining the Wave‑RNN model on your dataset, but eliminates the crunchy, low‑bit artefacts characteristic of insufficient quantization.

### What causes clicking noises at the end of generated audio clips?

Clicking stems from insufficient padding at upsampling layer boundaries in the vocoder. In [`vocoder/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/vocoder/hparams.py), raise `hp.voc_pad` from its default of `2` to `4` or `5`. Alternatively, verify that `tts_stop_threshold` in [`synthesizer/hparams.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/hparams.py) is not set too low, which can cause the synthesizer to truncate the mel‑spectrogram abruptly.

### How can I visualize the Tacotron alignment matrix to debug mispronunciations?

Import `plot_alignment` from `synthesizer.utils.plot` and call `synth.synthesize_spectrograms` with `return_alignments=True`. Save the resulting matrix using `plot_alignment(align[0], "debug.png")`. A healthy alignment shows a smooth diagonal line from top‑left to bottom‑right; gaps or jagged breaks indicate that the text‑to‑mel attention mechanism has failed to converge, usually requiring adjustment of `tts_stop_threshold` or additional training data.