How Voice Cloning and Custom Voices Work with Pocket TTS: Implementation Guide

Pocket TTS supports voice cloning by loading a "voice state" from preset embeddings, local audio files, or Hugging Face references to condition generation on reference audio prompts.

Voice cloning in the huggingface/speech-to-speech repository leverages the Kyutai Labs Pocket TTS model to generate speech that mimics specific speaker characteristics. The system achieves this through a stateful conditioning mechanism that extracts embeddings from reference audio and applies them consistently across streaming generation. Understanding how voice cloning and custom voices work with Pocket TTS requires examining the handler's setup pipeline and the voice state acquisition process.

Understanding Pocket TTS Voice Cloning Architecture

The Pocket TTS implementation treats voice identity as a conditioning state rather than a model parameter. When the handler initializes in src/speech_to_speech/TTS/pocket_tts_handler.py, it creates a persistent voice state object that contains the speaker embedding and related parameters extracted from your chosen reference.

The cloning workflow follows three distinct stages:

1. Model instantiation – The base TTS network loads via TTSModel.load_model()
2. Voice state acquisition – The system extracts embeddings from the reference audio using get_state_for_audio_prompt()
3. Stateful generation – The streaming inference reuses the voice state across chunks with copy_state=True to maintain timbre consistency

This architecture allows the same underlying model to generate unlimited distinct voices without retraining or fine-tuning.

Voice State Acquisition Methods

The pocket_tts_voice argument accepts three distinct input types, each resolving to a voice state object that conditions the generation.

Built-in Preset Voices

Pocket TTS ships with eight built-in voice embeddings that require no external files. The preset names include alba, marius, javert, jean, fantine, cosette, eponine, and azelma.

When you specify a preset name, the handler passes the string identifier directly to self.model.get_state_for_audio_prompt(), which retrieves the pre-computed embedding packaged with the model weights.

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="fantine",
    pocket_tts_device="cpu",
)

Local Audio Files for Custom Voices

For custom voice cloning, provide a path to a local WAV or PCM file containing clean speech samples. The handler automatically loads the file, resamples it if necessary, and extracts the speaker embedding during the setup phase.

The reference audio should be mono channel at approximately 16 kHz for optimal embedding extraction, though the system handles format conversion internally.

speech-to-speech \
    --pocket_tts_voice /path/to/my_voice.wav \
    --pocket_tts_device cuda

Hugging Face Hosted Voices

You can reference voices stored in Hugging Face repositories using the hf:// protocol. The handler downloads the specified file and processes it identically to local files.

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="hf://kyutai/tts-voices/custom_speaker",
    pocket_tts_device="mps",
)

The Voice Cloning Pipeline Implementation

The technical implementation in src/speech_to_speech/TTS/pocket_tts_handler.py handles voice state persistence and audio generation.

Setup and State Initialization

During the setup method (lines 71-86), the handler performs model loading and voice state acquisition:

from pocket_tts import TTSModel

# Load base model

self.model = TTSModel.load_model()

# Extract voice state from reference

self.voice_state = self.model.get_state_for_audio_prompt(voice)

The voice_state object encapsulates the speaker embedding and remains fixed for the handler's lifetime, ensuring consistent voice characteristics across all subsequent TTS calls.

Streaming Generation with State Preservation

During the process method (lines 54-59), the handler passes the voice state to the streaming generator:

for audio_chunk in self.model.generate_audio_stream(
    self.voice_state,
    text,
    max_tokens=self.max_tokens,
    copy_state=True,
):
    yield audio_chunk

The copy_state=True parameter creates an internal copy of the state for each generation cycle, preventing mutation of the original embedding and allowing safe reuse across multiple utterances.

Audio Resampling and Output Formatting

The model generates audio at 24 kHz, but the pipeline may require a different sample rate (default 16 kHz). The handler uses scipy.signal.resample_poly to convert between rates, calculating resampling factors from the greatest common divisor of the input and output rates (lines 41-47).

Audio chunks accumulate in a buffer until they contain at least blocksize samples, then convert to 16-bit PCM format (lines 71-99) before returning to the pipeline.

Practical Implementation Examples

Using Preset Voices Programmatically

from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="jean",
    pocket_tts_device="cuda",
)

audio_chunks = pipeline.run_tts("Hello, this is a preset voice demonstration.")
for chunk in audio_chunks:
    play(chunk)  # Your PCM playback implementation

Cloning from Local Custom Audio

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="/home/user/voice_samples/speaker_01.wav",
    pocket_tts_device="cuda",
)

for chunk in pipeline.run_tts("This speech uses my custom cloned voice."):
    stream_to_speaker(chunk)

Loading Voices from Hugging Face

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="hf://kyutai/tts-voices/celebrity_sample",
    pocket_tts_device="mps",
)

for chunk in pipeline.run_tts("Testing Hugging Face hosted voice cloning"):
    process_audio(chunk)

Summary

• Voice state architecture: Pocket TTS uses a stateful conditioning approach where get_state_for_audio_prompt() extracts embeddings from reference audio that persist across generation calls.
• Three input methods: Voice cloning supports preset names (alba, marius, javert, jean, fantine, cosette, eponine, azelma), local file paths, and Hugging Face references via the hf:// protocol.
• State preservation: The copy_state=True flag in generate_audio_stream() ensures voice embeddings remain immutable across multiple utterances.
• Automatic resampling: The handler converts the native 24 kHz output to the pipeline's configured sample rate using scipy.signal.resample_poly.
• Implementation location: Core logic resides in src/speech_to_speech/TTS/pocket_tts_handler.py, with CLI arguments defined in src/speech_to_speech/arguments_classes/pocket_tts_arguments.py.

Frequently Asked Questions

How do I prepare an audio file for custom voice cloning with Pocket TTS?

Prepare a mono-channel WAV or PCM file containing 10-30 seconds of clean, single-speaker speech at approximately 16 kHz. Background noise, music, or multiple speakers will degrade the embedding quality. Pass the absolute path to the --pocket_tts_voice argument or the pocket_tts_voice parameter in the pipeline constructor.

What is the difference between preset voices and custom voice files?

Preset voices (alba, marius, jean, etc.) are pre-computed embeddings bundled with the Kyutai Labs model weights that require no external files. Custom voice files trigger the embedding extraction pipeline during handler initialization, computing a voice state from your specific audio reference that captures unique speaker characteristics.

Can I switch voices without restarting the speech-to-speech pipeline?

No, the voice state initializes during the handler's setup method and remains fixed for the handler's lifetime. To change voices, you must instantiate a new SpeechToSpeechPipeline with a different pocket_tts_voice argument, as the voice state object is bound to the handler instance created in src/speech_to_speech/TTS/pocket_tts_handler.py.

Why does Pocket TTS output 24 kHz audio when my system uses 16 kHz?

The Kyutai Labs model natively generates high-fidelity audio at 24 kHz. The handler automatically resamples to your configured pipeline sample rate (default 16 kHz) using scipy.signal.resample_poly with factors derived from the greatest common divisor of the two rates. This occurs in the processing loop (lines 41-47) without requiring manual intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}