How to Evaluate Synthesized Speech Quality in Real-Time Voice Cloning: Objective Metrics and Subjective Methods

You can evaluate synthesized speech quality in the Real-Time-Voice-Cloning repository using objective metrics like Mel-Cepstral Distortion, speaker embedding cosine similarity, and external tools such as PESQ and STOI, alongside subjective methods like Mean Opinion Score (MOS) testing.

The Real-Time-Voice-Cloning repository by CorentinJ generates high-fidelity speech from text and a reference voice, but quantifying output quality requires systematic evaluation. To comprehensively evaluate synthesized speech quality, you must combine automated objective metrics that compare acoustic features against references with subjective listening tests that capture human perception of naturalness and similarity.

Objective Metrics Available in the Repository

Speaker Similarity via Encoder Embeddings

The repository ships a built-in speaker-verification encoder that produces 256-dimensional embeddings from audio. You can compare synthesized utterances against reference samples using cosine similarity to measure voice cloning accuracy.

In encoder/inference.py, the embed_utterance function extracts fixed-size embeddings from any wav file. Compute the cosine similarity between the reference embedding and the synthesized embedding to obtain a numerical similarity score ranging from -1 to 1, where higher values indicate better speaker match.

import numpy as np
from encoder import inference as encoder
from synthesizer.inference import Synthesizer
from pathlib import Path

# Load pretrained models

encoder.load_model(Path("saved_models/default/encoder.pt"))
synthesizer = Synthesizer(
    Path("saved_models/default/synthesizer.pt"),
    Path("saved_models/default/vocoder.pt"),
)

# Process reference and generate speech

ref_wav = Path("demo_voice.wav")
text = "The quick brown fox jumps over the lazy dog."
embed = encoder.embed_utterance(ref_wav)
generated_wav = synthesizer.synthesize_spectrograms([text], [embed])[0]

# Save and embed generated audio

generated_path = Path("generated.wav")
generated_path.write_bytes(generated_wav)
embed_gen = encoder.embed_utterance(generated_path)

# Calculate cosine similarity

cos_sim = np.dot(embed, embed_gen) / (np.linalg.norm(embed) * np.linalg.norm(embed_gen))
print(f"Speaker similarity (cosine): {cos_sim:.3f}")

Mel-Cepstral Distortion (MCD)

Mel-Cepstral Distortion measures the spectral distance between reference and generated speech by comparing mel-cepstral coefficients. Lower MCD values indicate higher acoustic similarity.

The synthesizer/audio.py module provides melspectrogram to extract mel-spectrograms from audio files. Use these to compute MCD with standard NumPy operations.

import numpy as np
from synthesizer import audio
from pathlib import Path

def mcd(mel_ref, mel_syn, sr=22050, n_mels=80):
    # mel shape: (n_mels, T)

    diff = mel_ref - mel_syn
    return (10.0 / np.log(10)) * np.sqrt(np.mean(diff ** 2))

# Load audio files

ref_path = Path("reference.wav")
gen_path = Path("generated.wav")

# Extract mel-spectrograms

mel_ref = audio.melspectrogram(ref_path.read_bytes(), sr=22050, num_mels=80)
mel_gen = audio.melspectrogram(gen_path.read_bytes(), sr=22050, num_mels=80)

print(f"MCD: {mcd(mel_ref, mel_gen):.2f} dB")

External Perceptual and Intelligibility Metrics

While not built into the core pipeline, you can export generated wav files and process them through specialized libraries:

• PESQ (Perceptual Evaluation of Speech Quality): Use pypesq to compute ITU-T P.862 scores that predict perceived quality for telephony speech.
• STOI (Short-Time Objective Intelligibility): Use pystoi to measure speech intelligibility, particularly useful for evaluating clarity in noisy conditions.
• Word Error Rate (WER): Run an external ASR model like Whisper on the synthesized audio and compare the transcription against the original text to measure intelligibility.

Export audio using the synthesis pipeline in synthesizer/inference.py or the demo scripts:

from synthesizer.inference import Synthesizer
from pathlib import Path

synthesizer = Synthesizer(
    Path("saved_models/default/synthesizer.pt"),
    Path("saved_models/default/vocoder.pt"),
)

text = "Hello world, this is a test of the voice cloning system."
embed = encoder.embed_utterance(Path("reference.wav"))
wav = synthesizer.synthesize_spectrograms([text], [embed])[0]

out = Path("evaluation_sample.wav")
out.write_bytes(wav)  # Feed to pypesq, pystoi, or ASR tools

Subjective Evaluation Methods

Mean Opinion Score (MOS)

Mean Opinion Score remains the gold standard for evaluating naturalness, where human listeners rate audio quality on a 1–5 scale. The repository does not include a built-in MOS interface, but you can export generated wav files from demo_toolbox.py or demo_cli.py for crowdsourced testing.

Alternatively, automate MOS prediction using MOSNet, a deep learning model that estimates MOS scores without human listeners. Process the exported evaluation_sample.wav through MOSNet to obtain predicted naturalness scores.

Diagnostic Visualization Tools

Attention Alignment Analysis

During training, the Tacotron synthesizer saves attention plots that serve as diagnostic quality indicators. Clear, diagonal attention patterns correlate with stable synthesis and proper text-to-speech alignment.

In synthesizer/train.py, the eval_model function generates these visualizations every tts_eval_interval steps, saving them to the plots/ subdirectory alongside wav and mel files.

from synthesizer.train import eval_model
from pathlib import Path
import numpy as np

# Assuming training tensors are available

eval_model(
    attention=attention_np,
    mel_prediction=mel_pred_np,
    target_spectrogram=target_spec_np,
    input_seq=text_np,
    step=10000,
    plot_dir=Path("plots/"),
    mel_output_dir=Path("mel/"),
    wav_dir=Path("wavs/"),
    sample_num=1,
    loss=loss_tensor,
    hparams=synthesizer.hparams,
)

This saves attention_step_10000_sample_1.png in the plots directory for manual inspection.

Summary

• Speaker similarity is computed using cosine similarity between 256-dimensional embeddings from encoder/inference.py.
• Mel-Cepstral Distortion utilizes synthesizer/audio.py to compare mel-spectrogram features between reference and generated audio.
• External metrics (PESQ, STOI, WER) require exporting wav files via synthesizer/inference.py and processing through specialized libraries.
• MOS evaluation can be performed manually on exported audio or automated using MOSNet.
• Attention visualization in synthesizer/train.py provides diagnostic insights into synthesis alignment quality.

Frequently Asked Questions

How do I measure speaker similarity in Real-Time-Voice-Cloning?

Use the embed_utterance function in encoder/inference.py to extract 256-dimensional embeddings from both the reference and synthesized audio, then calculate cosine similarity between the two vectors. Values closer to 1.0 indicate better voice cloning fidelity.

What is the best way to calculate Mel-Cepstral Distortion for generated audio?

Extract mel-spectrograms from both reference and synthesized files using audio.melspectrogram from synthesizer/audio.py, then apply the standard MCD formula comparing the mel-cepstral coefficients. Lower decibel values indicate higher acoustic similarity.

Can I use automated tools instead of human listeners for MOS evaluation?

Yes, while the repository supports manual MOS testing via exported wav files from demo_toolbox.py, you can automate naturalness scoring using MOSNet, a neural network trained to predict human MOS ratings from audio features without requiring live listeners.

Where are the attention plots saved during training?

The eval_model function in synthesizer/train.py saves attention visualizations to the plots/ subdirectory of your run directory every tts_eval_interval steps. These plots help diagnose text-to-speech alignment quality during the training process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}