# How Voice Cloning and Custom Voices Work with Pocket TTS: Implementation Guide

> Learn how Pocket TTS enables voice cloning and custom voices using preset embeddings, local files, or Hugging Face references. Implement your own unique speech generation today.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-10

---

**Pocket TTS supports voice cloning by loading a "voice state" from preset embeddings, local audio files, or Hugging Face references to condition generation on reference audio prompts.**

Voice cloning in the `huggingface/speech-to-speech` repository leverages the Kyutai Labs Pocket TTS model to generate speech that mimics specific speaker characteristics. The system achieves this through a stateful conditioning mechanism that extracts embeddings from reference audio and applies them consistently across streaming generation. Understanding how **voice cloning and custom voices work with Pocket TTS** requires examining the handler's setup pipeline and the voice state acquisition process.

## Understanding Pocket TTS Voice Cloning Architecture

The Pocket TTS implementation treats voice identity as a conditioning state rather than a model parameter. When the handler initializes in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py), it creates a persistent **voice state** object that contains the speaker embedding and related parameters extracted from your chosen reference.

The cloning workflow follows three distinct stages:

1. **Model instantiation** – The base TTS network loads via `TTSModel.load_model()`
2. **Voice state acquisition** – The system extracts embeddings from the reference audio using `get_state_for_audio_prompt()`
3. **Stateful generation** – The streaming inference reuses the voice state across chunks with `copy_state=True` to maintain timbre consistency

This architecture allows the same underlying model to generate unlimited distinct voices without retraining or fine-tuning.

## Voice State Acquisition Methods

The `pocket_tts_voice` argument accepts three distinct input types, each resolving to a voice state object that conditions the generation.

### Built-in Preset Voices

Pocket TTS ships with eight built-in voice embeddings that require no external files. The preset names include `alba`, `marius`, `javert`, `jean`, `fantine`, `cosette`, `eponine`, and `azelma`.

When you specify a preset name, the handler passes the string identifier directly to `self.model.get_state_for_audio_prompt()`, which retrieves the pre-computed embedding packaged with the model weights.

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="fantine",
    pocket_tts_device="cpu",
)

```

### Local Audio Files for Custom Voices

For custom voice cloning, provide a path to a local WAV or PCM file containing clean speech samples. The handler automatically loads the file, resamples it if necessary, and extracts the speaker embedding during the setup phase.

The reference audio should be mono channel at approximately 16 kHz for optimal embedding extraction, though the system handles format conversion internally.

```bash
speech-to-speech \
    --pocket_tts_voice /path/to/my_voice.wav \
    --pocket_tts_device cuda

```

### Hugging Face Hosted Voices

You can reference voices stored in Hugging Face repositories using the `hf://` protocol. The handler downloads the specified file and processes it identically to local files.

```python
pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="hf://kyutai/tts-voices/custom_speaker",
    pocket_tts_device="mps",
)

```

## The Voice Cloning Pipeline Implementation

The technical implementation in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) handles voice state persistence and audio generation.

### Setup and State Initialization

During the `setup` method (lines 71-86), the handler performs model loading and voice state acquisition:

```python
from pocket_tts import TTSModel

# Load base model

self.model = TTSModel.load_model()

# Extract voice state from reference

self.voice_state = self.model.get_state_for_audio_prompt(voice)

```

The `voice_state` object encapsulates the speaker embedding and remains fixed for the handler's lifetime, ensuring consistent voice characteristics across all subsequent TTS calls.

### Streaming Generation with State Preservation

During the `process` method (lines 54-59), the handler passes the voice state to the streaming generator:

```python
for audio_chunk in self.model.generate_audio_stream(
    self.voice_state,
    text,
    max_tokens=self.max_tokens,
    copy_state=True,
):
    yield audio_chunk

```

The `copy_state=True` parameter creates an internal copy of the state for each generation cycle, preventing mutation of the original embedding and allowing safe reuse across multiple utterances.

### Audio Resampling and Output Formatting

The model generates audio at 24 kHz, but the pipeline may require a different sample rate (default 16 kHz). The handler uses `scipy.signal.resample_poly` to convert between rates, calculating resampling factors from the greatest common divisor of the input and output rates (lines 41-47).

Audio chunks accumulate in a buffer until they contain at least `blocksize` samples, then convert to 16-bit PCM format (lines 71-99) before returning to the pipeline.

## Practical Implementation Examples

### Using Preset Voices Programmatically

```python
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="jean",
    pocket_tts_device="cuda",
)

audio_chunks = pipeline.run_tts("Hello, this is a preset voice demonstration.")
for chunk in audio_chunks:
    play(chunk)  # Your PCM playback implementation

```

### Cloning from Local Custom Audio

```python
pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="/home/user/voice_samples/speaker_01.wav",
    pocket_tts_device="cuda",
)

for chunk in pipeline.run_tts("This speech uses my custom cloned voice."):
    stream_to_speaker(chunk)

```

### Loading Voices from Hugging Face

```python
pipeline = SpeechToSpeechPipeline.from_pretrained(
    "speech-to-speech",
    pocket_tts_voice="hf://kyutai/tts-voices/celebrity_sample",
    pocket_tts_device="mps",
)

for chunk in pipeline.run_tts("Testing Hugging Face hosted voice cloning"):
    process_audio(chunk)

```

## Summary

- **Voice state architecture**: Pocket TTS uses a stateful conditioning approach where `get_state_for_audio_prompt()` extracts embeddings from reference audio that persist across generation calls.
- **Three input methods**: Voice cloning supports preset names (alba, marius, javert, jean, fantine, cosette, eponine, azelma), local file paths, and Hugging Face references via the `hf://` protocol.
- **State preservation**: The `copy_state=True` flag in `generate_audio_stream()` ensures voice embeddings remain immutable across multiple utterances.
- **Automatic resampling**: The handler converts the native 24 kHz output to the pipeline's configured sample rate using `scipy.signal.resample_poly`.
- **Implementation location**: Core logic resides in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py), with CLI arguments defined in [`src/speech_to_speech/arguments_classes/pocket_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/pocket_tts_arguments.py).

## Frequently Asked Questions

### How do I prepare an audio file for custom voice cloning with Pocket TTS?

Prepare a mono-channel WAV or PCM file containing 10-30 seconds of clean, single-speaker speech at approximately 16 kHz. Background noise, music, or multiple speakers will degrade the embedding quality. Pass the absolute path to the `--pocket_tts_voice` argument or the `pocket_tts_voice` parameter in the pipeline constructor.

### What is the difference between preset voices and custom voice files?

Preset voices (`alba`, `marius`, `jean`, etc.) are pre-computed embeddings bundled with the Kyutai Labs model weights that require no external files. Custom voice files trigger the embedding extraction pipeline during handler initialization, computing a voice state from your specific audio reference that captures unique speaker characteristics.

### Can I switch voices without restarting the speech-to-speech pipeline?

No, the voice state initializes during the handler's `setup` method and remains fixed for the handler's lifetime. To change voices, you must instantiate a new `SpeechToSpeechPipeline` with a different `pocket_tts_voice` argument, as the voice state object is bound to the handler instance created in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py).

### Why does Pocket TTS output 24 kHz audio when my system uses 16 kHz?

The Kyutai Labs model natively generates high-fidelity audio at 24 kHz. The handler automatically resamples to your configured pipeline sample rate (default 16 kHz) using `scipy.signal.resample_poly` with factors derived from the greatest common divisor of the two rates. This occurs in the processing loop (lines 41-47) without requiring manual intervention.