How the GE2E Similarity Matrix Is Computed in Real-Time Voice Cloning

The GE2E similarity matrix measures cosine similarity between utterance embeddings and speaker centroids, using exclusive centroids for same-speaker comparisons and inclusive centroids for different speakers to drive discriminative learning in the Real-Time-Voice-Cloning encoder.

The Generalized End-to-End (GE2E) loss is the training objective used in the CorentinJ/Real-Time-Voice-Cloning repository to learn speaker embeddings. At its core lies the GE2E similarity matrix, a 3-D tensor that quantifies how closely each utterance embedding aligns with every speaker centroid in a training batch. This matrix is computed in encoder/model.py and serves as the foundation for the contrastive learning mechanism that distinguishes between speakers.

Step-by-Step Computation in encoder/model.py

The SpeakerEncoder class constructs the similarity matrix through a four-stage process that implements the algorithm described in the GE2E paper. Each stage handles specific tensor operations to ensure utterances are compared against appropriate centroids.

Computing Inclusive Centroids

First, the algorithm calculates one centroid per speaker using all utterances from that speaker in the batch. This "inclusive" centroid represents the speaker's average embedding.

centroids_incl = torch.mean(embeds, dim=1, keepdim=True)
centroids_incl = centroids_incl.clone() / (torch.norm(centroids_incl, dim=2, keepdim=True) + 1e-5)

See lines 74–77 of encoder/model.py. The L2 normalization ensures the centroid is a unit vector, making subsequent dot products equivalent to cosine similarity.

Computing Exclusive Centroids

For each utterance, the algorithm computes an "exclusive" centroid that excludes the current utterance from the average. This prevents an utterance from being compared to itself during positive matching.

centroids_excl = (torch.sum(embeds, dim=1, keepdim=True) - embeds)
centroids_excl /= (utterances_per_speaker - 1)
centroids_excl = centroids_excl.clone() / (torch.norm(centroids_excl, dim=2, keepdim=True) + 1e-5)

See lines 78–82. By subtracting the current embedding from the sum before averaging, the centroid represents "all other utterances from this speaker."

Building the Similarity Matrix

The core computation constructs a 3-D tensor of shape (speakers_per_batch, utterances_per_speaker, speakers_per_batch). The matrix is populated using a masking strategy:

• Different speakers: The utterance is compared against the inclusive centroid of other speakers.
• Same speaker: The utterance is compared against its exclusive centroid.

sim_matrix = torch.zeros(speakers_per_batch, utterances_per_speaker,
                         speakers_per_batch).to(self.loss_device)
mask_matrix = 1 - np.eye(speakers_per_batch, dtype=np.int)
for j in range(speakers_per_batch):
    mask = np.where(mask_matrix[j])[0]
    sim_matrix[mask, :, j] = (embeds[mask] * centroids_incl[j]).sum(dim=2)
    sim_matrix[j, :, j] = (embeds[j] * centroids_excl[j]).sum(dim=1)

See lines 83–92. Because embeddings are L2-normalized, the dot product sum(dim=2) yields cosine similarity values between -1 and 1.

Scaling with Learnable Parameters

Before loss calculation, raw cosine similarities are transformed using learnable affine parameters similarity_weight and similarity_bias:

sim_matrix = sim_matrix * self.similarity_weight + self.similarity_bias

See line 104. This scaling allows the network to learn optimal decision boundaries for the softmax classification that follows.

The Role of the Similarity Matrix in Speaker Verification

The similarity matrix serves as the logits input to a softmax cross-entropy loss. After reshaping, each row represents an utterance and each column represents a potential speaker identity. The ground-truth label corresponds to the actual speaker who produced the utterance.

By using exclusive centroids for positive matches and inclusive centroids for negative matches, the loss function forces the encoder to:

• Pull embeddings closer to their speaker's centroid (excluding self-similarity).
• Push embeddings away from other speakers' centroids.

This contrastive optimization produces speaker-discriminative embeddings that generalize well to unseen speakers during voice cloning and verification tasks.

Practical Implementation Example

To compute the GE2E similarity matrix for a batch of embeddings:

import torch
from encoder.model import SpeakerEncoder

# Initialize encoder

encoder = SpeakerEncoder(device='cpu', loss_device='cpu')

# Dummy embeddings: (speakers, utterances, embedding_dim)

embeds = torch.randn(4, 5, 256)  # 4 speakers, 5 utterances each

# Normalize as done in forward()

embeds = embeds / (torch.norm(embeds, dim=2, keepdim=True) + 1e-5)

# Compute similarity matrix

sim_matrix = encoder.similarity_matrix(embeds)
print(sim_matrix.shape)   # → torch.Size([4, 5, 4])

To calculate the complete GE2E loss including the similarity matrix computation:

loss, eer = encoder.loss(embeds)
print(f'GE2E loss: {loss.item():.4f}, EER: {eer:.2%}')

These examples demonstrate the end-to-end flow from raw embeddings to the similarity matrix that drives training.

Summary

• The GE2E similarity matrix is computed in encoder/model.py within the SpeakerEncoder class.
• Inclusive centroids average all utterances per speaker for negative comparisons.
• Exclusive centroids exclude the current utterance for positive comparisons to avoid self-matching bias.
• The matrix uses cosine similarity (dot product of L2-normalized vectors) to measure embedding alignment.
• Learnable scaling parameters (similarity_weight and similarity_bias) optimize the decision boundary before softmax classification.
• The resulting matrix enables contrastive learning that produces robust speaker embeddings for voice cloning.

Frequently Asked Questions

What is the difference between inclusive and exclusive centroids in GE2E?

Inclusive centroids are the mean of all utterance embeddings for a given speaker in the batch, used when comparing utterances to other speakers. Exclusive centroids exclude the current utterance from the mean calculation, used when comparing an utterance to its own speaker's centroid. This distinction prevents the model from achieving trivially high similarity scores by matching an utterance to itself.

Why does the GE2E similarity matrix use cosine similarity rather than Euclidean distance?

The implementation uses cosine similarity (dot product) because the embeddings are L2-normalized to unit length. Cosine similarity measures directional alignment independent of vector magnitude, which is more appropriate for speaker verification where the angle between embeddings matters more than their absolute positions in the embedding space. The dot product (embeds * centroids).sum() provides an efficient computation of cosine similarity.

What is the purpose of the learnable weight and bias parameters in the similarity matrix?

According to the source code in encoder/model.py, the similarity_weight and similarity_bias parameters allow the network to learn optimal scaling and shifting of the raw cosine similarities before applying softmax. This learnable affine transformation acts similarly to a temperature parameter in contrastive learning, controlling the sharpness of the probability distribution and improving convergence during training.

How does the similarity matrix shape relate to batch composition?

The similarity matrix has shape (speakers_per_batch, utterances_per_speaker, speakers_per_batch), typically denoted as (N, M, N) where N is the number of speakers and M is utterances per speaker. This structure allows every utterance (represented by the first two dimensions) to be compared against every speaker centroid (the third dimension), creating a complete similarity graph across the batch for the GE2E loss calculation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}