# How the Encoder Creates Speaker Embeddings from Audio Inputs in Real-Time-Voice-Cloning

> Discover how the encoder generates speaker embeddings from audio inputs. Learn about waveform preprocessing, mel spectrogram conversion, LSTM processing, and L2 normalization for real-time voice cloning.

- Repository: [Corentin Jemine/Real-Time-Voice-Cloning](https://github.com/CorentinJ/Real-Time-Voice-Cloning)
- Tags: internals
- Published: 2026-03-05

---

**The encoder transforms raw audio into a fixed 256‑dimensional speaker embedding by preprocessing the waveform, converting it to a mel‑spectrogram, processing it through a 3‑layer LSTM with a linear projection, and applying L2 normalization.**

The speaker encoder is the cornerstone of the *Real-Time-Voice-Cloning* repository, responsible for distilling the unique vocal characteristics of any speaker into a compact vector representation. This embedding enables the subsequent synthesis modules to clone voices from mere seconds of audio. Understanding how the encoder creates speaker embeddings from audio inputs reveals the pipeline that bridges raw acoustic signals and deep neural network features.

## Audio Preprocessing Pipeline

Before neural processing begins, the raw waveform undergoes strict normalization and cleaning to ensure consistent input quality.

### Resampling and Volume Normalization

In [`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py), the `preprocess_wav()` function handles the initial transformation. It first resamples any input waveform to the target **16 kHz** sampling rate defined in [`encoder/params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_data.py) (`sampling_rate = 16000`) 【⁴³⁾】. The function then applies volume normalization using the `normalize_volume()` utility, targeting **-30 dBFS** (`audio_norm_target_dBFS`) to eliminate loudness variations across different recordings 【⁴⁵⁻⁴⁷⁾】.

### Voice Activity Detection and Trimming

The encoder employs WebRTC’s Voice Activity Detection (VAD) to strip silence from the audio. The `trim_long_silences()` function uses parameters specified in [`params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/params_data.py) (`vad_window_length = 30`, `vad_moving_average_width = 8`) to identify speech segments and remove non‑speech regions exceeding the configured thresholds 【⁴⁷⁻⁵⁰⁾】. This ensures the model processes only relevant acoustic content, improving embedding robustness.

## Mel-Spectrogram Feature Extraction

Once cleaned, the waveform becomes a time‑frequency representation suitable for the neural network.

The `wav_to_mel_spectrogram()` function in [`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py) computes a **40‑channel mel‑scale filter‑bank** spectrogram using 25 ms windows and 10 ms step sizes (`mel_window_length = 25.0`, `mel_window_step = 10.0`) 【⁵⁸⁻⁶⁴⁾】【⁶⁻⁹⁾】. The resulting matrix has shape `(n_mel_channels, n_frames)`, which the encoder transposes to `(n_frames, n_channels)` to create a temporal sequence where each time step represents 10 ms of audio 【⁶⁴⁻⁶⁵⁾】.

## Neural Network Architecture

The `SpeakerEncoder` class in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py) implements the core embedding network using recurrent and projection layers.

### LSTM Processing

The model contains a **3‑layer LSTM** (`model_num_layers = 3`) with a hidden size of 256 (`model_hidden_size = 256`) 【¹⁸⁻²¹⁾】【²³⁻²⁴⁾】. The mel‑spectrogram frames feed sequentially into `self.lstm`, which processes the temporal dynamics of the speech signal. The architecture extracts only the **final hidden state** from the last LSTM layer, discarding the output sequence to capture a summary vector of the entire utterance.

### Linear Projection and L2 Normalization

The LSTM hidden state passes through a linear layer (`self.linear`) followed by a **ReLU activation**, producing a raw embedding vector of size 256 (`model_embedding_size = 256`) 【²²⁻²⁴⁾】. Critically, the encoder applies **L2 normalization** to constrain the vector to unit length, yielding the final speaker embedding 【⁵⁵⁻⁶⁰⁾】. This normalization ensures that similarity comparisons rely purely on angular distance rather than vector magnitude.

## Inference Strategies for Variable Lengths

Real‑world utterances vary in duration, so [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) implements a **partial‑utterance strategy** to maintain consistent embedding quality.

The `embed_utterance()` function can split long audio into overlapping partial segments using `compute_partial_slices()`, typically targeting **1.6 seconds** of audio per partial (160 frames at 100 fps). Each partial generates an independent embedding via `embed_frames_batch()`, and the final utterance embedding is the **mean of all partial embeddings**, re‑normalized to unit length 【⁴⁸⁻⁵¹⁾】. This approach prevents performance degradation on short clips while handling longer recordings gracefully.

## Training vs. Inference Differences

During training, the encoder uses an additional **similarity scaling mechanism**. The model learns a weight and bias (`self.similarity_weight`, `self.similarity_bias`) that scale cosine similarities between embeddings before computing the Generalized End‑to‑End (GE2E) loss 【²⁶⁻²⁹⁾】. These parameters adjust the sharpness of the similarity distribution but are **not used during inference**, where only the normalized embedding matters 【³¹⁻³³⁾】.

## Practical Code Examples

Load a pretrained encoder and generate embeddings from an audio file:

```python
from pathlib import Path
from encoder import inference as enc
import soundfile as sf

# Initialize the encoder

weights_path = Path("pretrained/encoder.pt")
enc.load_model(weights_path)

# Load and preprocess audio

wav, source_sr = sf.read("speaker_sample.wav")
wav = enc.preprocess_wav(wav, source_sr=source_sr)

# Generate embedding (shape: (256,))

embedding = enc.embed_utterance(wav)
print(f"Embedding shape: {embedding.shape}, norm: {embedding.norm():.4f}")

```

Process audio with partial utterances for robustness:

```python

# Returns the averaged embedding and individual partial embeddings

embed, partial_embeds, _ = enc.embed_utterance(
    wav, 
    using_partials=True, 
    return_partials=True
)
print(f"Generated {len(partial_embeds)} partial embeddings")

```

## Summary

- The encoder pipeline follows a strict sequence: **waveform → resampling/VAD → mel‑spectrogram → 3‑layer LSTM → linear projection → ReLU → L2 normalization**.
- Key hyperparameters (16 kHz sample rate, 40 mel channels, 256‑dimensional embeddings) are defined in [`encoder/params_data.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/params_data.py).
- The `SpeakerEncoder` class in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py) uses the final LSTM hidden state and enforces unit‑norm embeddings for cosine‑similarity comparisons.
- [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) handles variable‑length audio through partial utterance averaging, ensuring consistent embedding quality regardless of input duration.

## Frequently Asked Questions

### What audio format does the encoder require?

The encoder accepts raw PCM waveforms as NumPy arrays of type `float32`. While [`encoder/audio.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/audio.py) handles resampling automatically, providing audio at **16 kHz** minimizes preprocessing overhead. Mono channel audio is required; stereo inputs should be converted to mono before processing.

### Why does the encoder use L2 normalization on embeddings?

L2 normalization constrains all embeddings to the surface of a unit hypersphere. According to the implementation in [`encoder/model.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/model.py), this ensures that speaker similarity is measured purely by the **angle between vectors** (cosine similarity) rather than magnitude, making the embedding space geometrically consistent for voice cloning tasks.

### How long must an audio sample be to generate a reliable embedding?

The encoder can work with as little as **1.6 seconds** of speech, which corresponds to 160 mel frames. For shorter clips, [`encoder/inference.py`](https://github.com/CorentinJ/Real-Time-Voice-Cloning/blob/main/encoder/inference.py) disables partial utterances and processes the full available audio. For optimal results, the repository recommends **5–10 seconds** of clean speech to capture sufficient speaker characteristics.

### What is the difference between the GE2E loss and the inference embedding?

During training, the GE2E loss uses learnable similarity scaling parameters to sharpen the distinction between speaker centroids. However, at inference time, the encoder discards these scaling factors and outputs only the **L2‑normalized embedding vector**. This means the similarity weights affect how the model learns but do not alter the final embedding geometry used for voice cloning.