# How the GE2E Similarity Matrix Is Computed in Real-Time Voice Cloning

> Learn how the GE2E similarity matrix computes cosine similarity to drive discriminative learning in real-time voice cloning. Understand its role in speaker embedding.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: deep-dive
- Published: 2026-03-05

---

**The GE2E similarity matrix measures cosine similarity between utterance embeddings and speaker centroids, using exclusive centroids for same-speaker comparisons and inclusive centroids for different speakers to drive discriminative learning in the Real-Time-Voice-Cloning encoder.**

The Generalized End-to-End (GE2E) loss is the training objective used in the [CorentinJ/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning) repository to learn speaker embeddings. At its core lies the **GE2E similarity matrix**, a 3-D tensor that quantifies how closely each utterance embedding aligns with every speaker centroid in a training batch. This matrix is computed in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py) and serves as the foundation for the contrastive learning mechanism that distinguishes between speakers.

## Step-by-Step Computation in encoder/model.py

The `SpeakerEncoder` class constructs the similarity matrix through a four-stage process that implements the algorithm described in the GE2E paper. Each stage handles specific tensor operations to ensure utterances are compared against appropriate centroids.

### Computing Inclusive Centroids

First, the algorithm calculates one centroid per speaker using **all** utterances from that speaker in the batch. This "inclusive" centroid represents the speaker's average embedding.

```python
centroids_incl = torch.mean(embeds, dim=1, keepdim=True)
centroids_incl = centroids_incl.clone() / (torch.norm(centroids_incl, dim=2, keepdim=True) + 1e-5)

```

*See lines 74–77 of [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py).* The L2 normalization ensures the centroid is a unit vector, making subsequent dot products equivalent to cosine similarity.

### Computing Exclusive Centroids

For each utterance, the algorithm computes an "exclusive" centroid that excludes the current utterance from the average. This prevents an utterance from being compared to itself during positive matching.

```python
centroids_excl = (torch.sum(embeds, dim=1, keepdim=True) - embeds)
centroids_excl /= (utterances_per_speaker - 1)
centroids_excl = centroids_excl.clone() / (torch.norm(centroids_excl, dim=2, keepdim=True) + 1e-5)

```

*See lines 78–82.* By subtracting the current embedding from the sum before averaging, the centroid represents "all other utterances from this speaker."

### Building the Similarity Matrix

The core computation constructs a 3-D tensor of shape `(speakers_per_batch, utterances_per_speaker, speakers_per_batch)`. The matrix is populated using a masking strategy:

- **Different speakers**: The utterance is compared against the **inclusive** centroid of other speakers.
- **Same speaker**: The utterance is compared against its **exclusive** centroid.

```python
sim_matrix = torch.zeros(speakers_per_batch, utterances_per_speaker,
                         speakers_per_batch).to(self.loss_device)
mask_matrix = 1 - np.eye(speakers_per_batch, dtype=np.int)
for j in range(speakers_per_batch):
    mask = np.where(mask_matrix[j])[0]
    sim_matrix[mask, :, j] = (embeds[mask] * centroids_incl[j]).sum(dim=2)
    sim_matrix[j, :, j] = (embeds[j] * centroids_excl[j]).sum(dim=1)

```

*See lines 83–92.* Because embeddings are L2-normalized, the dot product `sum(dim=2)` yields cosine similarity values between -1 and 1.

### Scaling with Learnable Parameters

Before loss calculation, raw cosine similarities are transformed using learnable affine parameters `similarity_weight` and `similarity_bias`:

```python
sim_matrix = sim_matrix * self.similarity_weight + self.similarity_bias

```

*See line 104.* This scaling allows the network to learn optimal decision boundaries for the softmax classification that follows.

## The Role of the Similarity Matrix in Speaker Verification

The similarity matrix serves as the logits input to a softmax cross-entropy loss. After reshaping, each row represents an utterance and each column represents a potential speaker identity. The ground-truth label corresponds to the actual speaker who produced the utterance.

By using **exclusive centroids** for positive matches and **inclusive centroids** for negative matches, the loss function forces the encoder to:

- Pull embeddings closer to their speaker's centroid (excluding self-similarity).
- Push embeddings away from other speakers' centroids.

This contrastive optimization produces speaker-discriminative embeddings that generalize well to unseen speakers during voice cloning and verification tasks.

## Practical Implementation Example

To compute the GE2E similarity matrix for a batch of embeddings:

```python
import torch
from encoder.model import SpeakerEncoder

# Initialize encoder

encoder = SpeakerEncoder(device='cpu', loss_device='cpu')

# Dummy embeddings: (speakers, utterances, embedding_dim)

embeds = torch.randn(4, 5, 256)  # 4 speakers, 5 utterances each

# Normalize as done in forward()

embeds = embeds / (torch.norm(embeds, dim=2, keepdim=True) + 1e-5)

# Compute similarity matrix

sim_matrix = encoder.similarity_matrix(embeds)
print(sim_matrix.shape)   # → torch.Size([4, 5, 4])

```

To calculate the complete GE2E loss including the similarity matrix computation:

```python
loss, eer = encoder.loss(embeds)
print(f'GE2E loss: {loss.item():.4f}, EER: {eer:.2%}')

```

These examples demonstrate the end-to-end flow from raw embeddings to the similarity matrix that drives training.

## Summary

- The **GE2E similarity matrix** is computed in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py) within the `SpeakerEncoder` class.
- **Inclusive centroids** average all utterances per speaker for negative comparisons.
- **Exclusive centroids** exclude the current utterance for positive comparisons to avoid self-matching bias.
- The matrix uses **cosine similarity** (dot product of L2-normalized vectors) to measure embedding alignment.
- **Learnable scaling parameters** (`similarity_weight` and `similarity_bias`) optimize the decision boundary before softmax classification.
- The resulting matrix enables **contrastive learning** that produces robust speaker embeddings for voice cloning.

## Frequently Asked Questions

### What is the difference between inclusive and exclusive centroids in GE2E?

**Inclusive centroids** are the mean of all utterance embeddings for a given speaker in the batch, used when comparing utterances to *other* speakers. **Exclusive centroids** exclude the current utterance from the mean calculation, used when comparing an utterance to its *own* speaker's centroid. This distinction prevents the model from achieving trivially high similarity scores by matching an utterance to itself.

### Why does the GE2E similarity matrix use cosine similarity rather than Euclidean distance?

The implementation uses **cosine similarity** (dot product) because the embeddings are L2-normalized to unit length. Cosine similarity measures directional alignment independent of vector magnitude, which is more appropriate for speaker verification where the angle between embeddings matters more than their absolute positions in the embedding space. The dot product `(embeds * centroids).sum()` provides an efficient computation of cosine similarity.

### What is the purpose of the learnable weight and bias parameters in the similarity matrix?

According to the source code in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py), the `similarity_weight` and `similarity_bias` parameters allow the network to learn optimal scaling and shifting of the raw cosine similarities before applying softmax. This learnable affine transformation acts similarly to a temperature parameter in contrastive learning, controlling the sharpness of the probability distribution and improving convergence during training.

### How does the similarity matrix shape relate to batch composition?

The similarity matrix has shape `(speakers_per_batch, utterances_per_speaker, speakers_per_batch)`, typically denoted as `(N, M, N)` where *N* is the number of speakers and *M* is utterances per speaker. This structure allows every utterance (represented by the first two dimensions) to be compared against every speaker centroid (the third dimension), creating a complete similarity graph across the batch for the GE2E loss calculation.