Real-Time Voice Cloning Vocoder: Audio Sampling Rates and Formats Compatibility

The vocoder in CorentinJ/Real-Time-Voice-Cloning operates at a default 16 kHz sampling rate and supports any audio format readable by librosa (including WAV, FLAC, OGG, and MP3), outputting 16-bit PCM WAV files.

Understanding the audio sampling rates and formats compatible with the vocoder is essential for preprocessing training data and generating high-quality cloned speech. According to the Real-Time-Voice-Cloning source code, the vocoder inherits its audio processing parameters from the synthesizer module and relies on standard Python audio libraries for I/O operations.

Default Sampling Rate Configuration

The vocoder expects raw audio sampled at 16 kHz by default. This value originates in synthesizer/hparams.py, where sample_rate is explicitly set to 16000 Hz at line 23. The vocoder module imports these hyperparameters in vocoder/hparams.py at line 6, ensuring both components operate on identical audio timelines.

You can modify this rate by editing hp.sample_rate in vocoder/hparams.py. However, the mel-spectrogram generation and neural network inference assume consistent timing parameters. Changing the sampling rate without retraining the model or adjusting dependent parameters (frame length, hop length) will cause dimension mismatches. Therefore, 16 kHz remains the safest and recommended configuration.

Supported Audio Formats

Input file compatibility is determined by the loading mechanism in vocoder/audio.py. The load_wav() function (lines 19–21) utilizes librosa.load(), which internally delegates to soundfile (libsndfile) and audioread backends.

As implemented in CorentinJ/Real-Time-Voice-Cloning, this pipeline supports the following formats:

• PCM WAV (.wav)
• FLAC (.flac)
• OGG/Vorbis (.ogg)
• MP3 (.mp3)
• Other formats readable by soundfile or audioread (including AIFF, AU, and CAF)

librosa.load() automatically resamples incoming audio to the target rate specified in hp.sample_rate during the loading process, standardizing inputs regardless of the source file's original sampling rate.

Audio Output Specifications

When saving synthesized waveforms, the vocoder uses sf.write() from the soundfile library (lines 24–25 in vocoder/audio.py). This implementation produces 16-bit PCM WAV files at the configured sampling rate. The output bit depth and format are hardcoded in the save routine to ensure compatibility with standard media players and downstream processing tools.

Practical Implementation

The following example demonstrates loading an input file (any supported format) and saving the vocoder output:

import vocoder.audio as voc_audio
import vocoder.hparams as hp

# Load an audio file and automatically resample to 16 kHz

wav, sr = voc_audio.load_wav("input.mp3")  # sr == hp.sample_rate == 16000

# Process the waveform through the vocoder model

# (mel-spectrogram inference and waveform generation steps here)

# Save the generated waveform as a 16-bit PCM WAV file

voc_audio.save_wav(wav, "output.wav")

Summary

• The vocoder defaults to 16 kHz sampling rate, inherited from synthesizer/hparams.py and referenced in vocoder/hparams.py.
• Input formats include WAV, FLAC, OGG, MP3, and any format supported by librosa/soundfile via vocoder/audio.py.
• Output is always a 16-bit PCM WAV file written using soundfile.write().
• Modifying hp.sample_rate requires careful adjustment of related hyperparameters to avoid inference errors.

Frequently Asked Questions

Can I use audio with a different sampling rate than 16 kHz?

Yes, but librosa.load() in vocoder/audio.py automatically resamples input files to 16 kHz during loading. To use a different rate natively, you must change hp.sample_rate in vocoder/hparams.py and ensure the synthesizer and vocoder models are trained or fine-tuned with matching parameters.

Does the vocoder support MP3 files directly?

Yes. The vocoder supports MP3 files because librosa.load() leverages audioread and soundfile backends, which handle MP3 decoding. You can pass .mp3 files directly to voc_audio.load_wav() without manual conversion.

What bit depth does the vocoder output?

The vocoder outputs 16-bit PCM WAV files. This is determined by the sf.write() call in vocoder/audio.py (line 24), which uses soundfile's default PCM_16 subtype when writing the synthesized waveform.

Where do I change the vocoder sampling rate?

Change the sample_rate value in vocoder/hparams.py (line 6), which imports from synthesizer/hparams.py. Ensure this value matches the sampling rate used during model training and mel-spectrogram extraction to prevent runtime dimension errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}