# How to Evaluate Synthesized Speech Quality in Real-Time Voice Cloning: Objective Metrics and Subjective Methods

> Evaluate synthesized speech quality in Real-Time Voice Cloning using objective metrics like MCD PESQ STOI and subjective MOS tests. Learn how to measure naturalness and speaker similarity effectively.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: deep-dive
- Published: 2026-03-05

---

**You can evaluate synthesized speech quality in the Real-Time-Voice-Cloning repository using objective metrics like Mel-Cepstral Distortion, speaker embedding cosine similarity, and external tools such as PESQ and STOI, alongside subjective methods like Mean Opinion Score (MOS) testing.**

The Real-Time-Voice-Cloning repository by CorentinJ generates high-fidelity speech from text and a reference voice, but quantifying output quality requires systematic evaluation. To comprehensively evaluate synthesized speech quality, you must combine automated objective metrics that compare acoustic features against references with subjective listening tests that capture human perception of naturalness and similarity.

## Objective Metrics Available in the Repository

### Speaker Similarity via Encoder Embeddings

The repository ships a built-in **speaker-verification encoder** that produces 256-dimensional embeddings from audio. You can compare synthesized utterances against reference samples using **cosine similarity** to measure voice cloning accuracy.

In [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py), the `embed_utterance` function extracts fixed-size embeddings from any wav file. Compute the cosine similarity between the reference embedding and the synthesized embedding to obtain a numerical similarity score ranging from -1 to 1, where higher values indicate better speaker match.

```python
import numpy as np
from encoder import inference as encoder
from synthesizer.inference import Synthesizer
from pathlib import Path

# Load pretrained models

encoder.load_model(Path("saved_models/default/encoder.pt"))
synthesizer = Synthesizer(
    Path("saved_models/default/synthesizer.pt"),
    Path("saved_models/default/vocoder.pt"),
)

# Process reference and generate speech

ref_wav = Path("demo_voice.wav")
text = "The quick brown fox jumps over the lazy dog."
embed = encoder.embed_utterance(ref_wav)
generated_wav = synthesizer.synthesize_spectrograms([text], [embed])[0]

# Save and embed generated audio

generated_path = Path("generated.wav")
generated_path.write_bytes(generated_wav)
embed_gen = encoder.embed_utterance(generated_path)

# Calculate cosine similarity

cos_sim = np.dot(embed, embed_gen) / (np.linalg.norm(embed) * np.linalg.norm(embed_gen))
print(f"Speaker similarity (cosine): {cos_sim:.3f}")

```

### Mel-Cepstral Distortion (MCD)

**Mel-Cepstral Distortion** measures the spectral distance between reference and generated speech by comparing mel-cepstral coefficients. Lower MCD values indicate higher acoustic similarity.

The [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py) module provides `melspectrogram` to extract mel-spectrograms from audio files. Use these to compute MCD with standard NumPy operations.

```python
import numpy as np
from synthesizer import audio
from pathlib import Path

def mcd(mel_ref, mel_syn, sr=22050, n_mels=80):
    # mel shape: (n_mels, T)

    diff = mel_ref - mel_syn
    return (10.0 / np.log(10)) * np.sqrt(np.mean(diff ** 2))

# Load audio files

ref_path = Path("reference.wav")
gen_path = Path("generated.wav")

# Extract mel-spectrograms

mel_ref = audio.melspectrogram(ref_path.read_bytes(), sr=22050, num_mels=80)
mel_gen = audio.melspectrogram(gen_path.read_bytes(), sr=22050, num_mels=80)

print(f"MCD: {mcd(mel_ref, mel_gen):.2f} dB")

```

### External Perceptual and Intelligibility Metrics

While not built into the core pipeline, you can export generated wav files and process them through specialized libraries:

- **PESQ (Perceptual Evaluation of Speech Quality)**: Use `pypesq` to compute ITU-T P.862 scores that predict perceived quality for telephony speech.
- **STOI (Short-Time Objective Intelligibility)**: Use `pystoi` to measure speech intelligibility, particularly useful for evaluating clarity in noisy conditions.
- **Word Error Rate (WER)**: Run an external ASR model like Whisper on the synthesized audio and compare the transcription against the original text to measure intelligibility.

Export audio using the synthesis pipeline in [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) or the demo scripts:

```python
from synthesizer.inference import Synthesizer
from pathlib import Path

synthesizer = Synthesizer(
    Path("saved_models/default/synthesizer.pt"),
    Path("saved_models/default/vocoder.pt"),
)

text = "Hello world, this is a test of the voice cloning system."
embed = encoder.embed_utterance(Path("reference.wav"))
wav = synthesizer.synthesize_spectrograms([text], [embed])[0]

out = Path("evaluation_sample.wav")
out.write_bytes(wav)  # Feed to pypesq, pystoi, or ASR tools

```

## Subjective Evaluation Methods

### Mean Opinion Score (MOS)

**Mean Opinion Score** remains the gold standard for evaluating naturalness, where human listeners rate audio quality on a 1–5 scale. The repository does not include a built-in MOS interface, but you can export generated wav files from [`demo_toolbox.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_toolbox.py) or [`demo_cli.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_cli.py) for crowdsourced testing.

Alternatively, automate MOS prediction using **MOSNet**, a deep learning model that estimates MOS scores without human listeners. Process the exported `evaluation_sample.wav` through MOSNet to obtain predicted naturalness scores.

## Diagnostic Visualization Tools

### Attention Alignment Analysis

During training, the Tacotron synthesizer saves attention plots that serve as diagnostic quality indicators. Clear, diagonal attention patterns correlate with stable synthesis and proper text-to-speech alignment.

In [`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py), the `eval_model` function generates these visualizations every `tts_eval_interval` steps, saving them to the `plots/` subdirectory alongside wav and mel files.

```python
from synthesizer.train import eval_model
from pathlib import Path
import numpy as np

# Assuming training tensors are available

eval_model(
    attention=attention_np,
    mel_prediction=mel_pred_np,
    target_spectrogram=target_spec_np,
    input_seq=text_np,
    step=10000,
    plot_dir=Path("plots/"),
    mel_output_dir=Path("mel/"),
    wav_dir=Path("wavs/"),
    sample_num=1,
    loss=loss_tensor,
    hparams=synthesizer.hparams,
)

```

This saves `attention_step_10000_sample_1.png` in the plots directory for manual inspection.

## Summary

- **Speaker similarity** is computed using cosine similarity between 256-dimensional embeddings from [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py).
- **Mel-Cepstral Distortion** utilizes [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py) to compare mel-spectrogram features between reference and generated audio.
- **External metrics** (PESQ, STOI, WER) require exporting wav files via [`synthesizer/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/inference.py) and processing through specialized libraries.
- **MOS evaluation** can be performed manually on exported audio or automated using MOSNet.
- **Attention visualization** in [`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py) provides diagnostic insights into synthesis alignment quality.

## Frequently Asked Questions

### How do I measure speaker similarity in Real-Time-Voice-Cloning?

Use the `embed_utterance` function in [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) to extract 256-dimensional embeddings from both the reference and synthesized audio, then calculate cosine similarity between the two vectors. Values closer to 1.0 indicate better voice cloning fidelity.

### What is the best way to calculate Mel-Cepstral Distortion for generated audio?

Extract mel-spectrograms from both reference and synthesized files using `audio.melspectrogram` from [`synthesizer/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/audio.py), then apply the standard MCD formula comparing the mel-cepstral coefficients. Lower decibel values indicate higher acoustic similarity.

### Can I use automated tools instead of human listeners for MOS evaluation?

Yes, while the repository supports manual MOS testing via exported wav files from [`demo_toolbox.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/demo_toolbox.py), you can automate naturalness scoring using MOSNet, a neural network trained to predict human MOS ratings from audio features without requiring live listeners.

### Where are the attention plots saved during training?

The `eval_model` function in [`synthesizer/train.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/synthesizer/train.py) saves attention visualizations to the `plots/` subdirectory of your run directory every `tts_eval_interval` steps. These plots help diagnose text-to-speech alignment quality during the training process.