# How to Implement Voice Cloning Using Pocket TTS in the Speech-to-Speech Pipeline

> Learn to implement voice cloning with Pocket TTS in the Hugging Face Speech-to-Speech pipeline. Extract speaker embeddings from audio with PocketTTSHandler for seamless integration.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-05

---

**Pocket TTS enables voice cloning in the Hugging Face Speech-to-Speech pipeline by extracting speaker embeddings from reference audio via the `PocketTTSHandler`, supporting preset voices, local files, or Hugging Face repositories.**

Voice cloning transforms text-to-speech systems by replicating a specific speaker's characteristics. In the `huggingface/speech-to-speech` repository, **Pocket TTS** serves as the lightweight, streaming-enabled backend that makes this possible. This guide walks through the complete implementation, from CLI usage to programmatic integration, based on the actual source code architecture.

## What is Pocket TTS?

Pocket TTS is a subclass of `BaseHandler` located in [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py). It converts LLM responses into audio streams while supporting **zero-shot voice cloning**—the ability to mimic a new speaker from a short audio sample without model retraining.

The handler operates in a streaming fashion, yielding 16 kHz `int16` audio blocks that downstream components consume in real time.

## How Voice Cloning Works in Pocket TTS

The cloning mechanism follows four distinct phases implemented in the handler's lifecycle:

1. **Voice specification** — The `voice` parameter accepts preset names (e.g., `alba`, `jean`), local file paths, or Hugging Face references (`hf://...`).

2. **Model initialization** — In `setup()` (lines 66-73), `TTSModel.load_model()` loads the weights and moves them to the target device (`cpu`, `cuda`, or `mps`).

3. **Speaker embedding extraction** — The critical cloning step occurs at lines 83-86: `self.model.get_state_for_audio_prompt(voice)` computes speaker embeddings from the reference audio, creating a reusable voice state.

4. **Streaming synthesis** — During `process()` (lines 98-127), `generate_audio_stream()` uses the cached voice state to synthesize speech that matches the target speaker's timbre.

## Installation and Setup

Before implementing voice cloning, install the optional Pocket TTS dependency:

```bash
pip install "speech-to-speech[pocket]"

```

This installs the underlying Moshi/TTS libraries required for model inference and speaker conditioning.

## CLI Implementation: Three Voice Source Options

The pipeline accepts voices from three distinct sources, all controlled via `--pocket_tts_voice`.

### Option 1: Built-in Preset Voices

```bash
python -m speech_to_speech.main \
    --tts pocket \
    --pocket_tts_voice alba \
    --pocket_tts_device cuda

```

Presets like `alba` and `jean` are bundled with the package and require no external files.

### Option 2: Local Audio File

```bash
python -m speech_to_speech.main \
    --tts pocket \
    --pocket_tts_voice ./samples/alice.wav \
    --pocket_tts_device cuda \
    --pocket_tts_sample_rate 16000 \
    --pocket_tts_blocksize 512

```

The reference file should contain clean, single-speaker speech. Supported formats include WAV and OGG.

### Option 3: Hugging Face Repository

```bash
python -m speech_to_speech.main \
    --tts pocket \
    --pocket_tts_voice hf://kyutai/tts-voices/voice-en-frank \
    --pocket_tts_device cpu

```

The `hf://` prefix triggers automatic download and caching from the Hugging Face Hub.

## Programmatic Implementation

For custom pipelines, instantiate `PocketTTSHandler` directly and feed it `TTSIn` messages.

### Minimal Working Example

```python
import queue
from threading import Event
from speech_to_speech.TTS.pocket_tts_handler import PocketTTSHandler
from speech_to_speech.pipeline.handler_types import TTSIn

# Synchronization primitive required by the handler

should_listen = Event()
should_listen.set()

handler = PocketTTSHandler(
    should_listen,
    device="cpu",
    voice="hf://kyutai/tts-voices/voice-en-sarah",
    sample_rate=16000,
    blocksize=512,
    max_tokens=100,
)

# Process text and receive streaming audio blocks

for audio_block in handler.process(
    TTSIn(text="This cloned voice maintains consistent speaker characteristics.")
):
    # audio_block: NumPy int16 array at 16 kHz

    print(f"Received {len(audio_block)} samples")

```

### Advanced: Manual Pipeline Assembly

```python
from threading import Event
from speech_to_speech.TTS.pocket_tts_handler import PocketTTSHandler

should_listen = Event()

handler = PocketTTSHandler(
    should_listen,
    device="cuda",
    voice="/path/to/my_voice.wav",  # Local file cloning

    sample_rate=16000,
    blocksize=512,
    max_tokens=50,
)

# Iterate over streamed synthesis results

for audio_chunk in handler.process(TTSIn(text="Hello, this is my cloned voice!")):
    # Route to audio sink: file, websocket, or playback

    stream_to_output(audio_chunk)

```

## Pipeline Integration Architecture

Understanding how `PocketTTSHandler` fits into the broader system clarifies customization points.

### Argument Mapping

`PocketTTSHandlerArguments` in [`src/speech_to_speech/arguments_classes/pocket_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/pocket_tts_arguments.py) bridges CLI flags to constructor parameters:

| CLI Flag | Handler Parameter | Default |
|----------|-------------------|---------|
| `--pocket_tts_device` | `device` | `"cpu"` |
| `--pocket_tts_voice` | `voice` | `"alba"` |
| `--pocket_tts_sample_rate` | `sample_rate` | `16000` |
| `--pocket_tts_blocksize` | `blocksize` | `512` |
| `--pocket_tts_max_tokens` | `max_tokens` | `50` |

### Handler Factory

In [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 106-115), the `get_tts_handler` factory routes `--tts pocket` to `PocketTTSHandler`:

```python

# From s2s_pipeline.py

if module_kwargs.tts == "pocket":
    from .TTS.pocket_tts_handler import PocketTTSHandler
    return PocketTTSHandler(**pocket_tts_kwargs)

```

This factory pattern enables swapping TTS backends without changing downstream code.

### Data Flow

```

LLM response (TTSIn)
    ↓
PocketTTSHandler.process()
    ↓
generate_audio_stream() with voice state
    ↓
int16 audio blocks (16 kHz)
    ↓
LocalAudioStreamer / WebSocketStreamer
    ↓
Playback or network transmission

```

## Performance and Quality Considerations

- **Device selection**: GPU (`cuda`) reduces latency significantly for real-time applications; `mps` is available for Apple Silicon.

- **Block size**: Smaller `blocksize` values (e.g., 256) reduce latency but increase per-chunk overhead. The default 512 samples balances responsiveness and efficiency.

- **Reference audio quality**: Clean, noise-free samples of 3-10 seconds yield optimal cloning. The model in [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) extracts embeddings robustly across languages.

- **Max tokens**: Limits LLM response length processed per synthesis call. Increase for longer utterances, decrease for faster turn-taking.

## Key Source Files

| Path | Purpose |
|------|---------|
| [`src/speech_to_speech/TTS/pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/pocket_tts_handler.py) | Core handler with `setup()` and `process()` methods |
| [`src/speech_to_speech/arguments_classes/pocket_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/pocket_tts_arguments.py) | CLI-to-constructor argument mapping |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Handler factory at lines 106-115 |
| [`src/speech_to_speech/connections/local_audio_streamer.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/connections/local_audio_streamer.py) | Audio block consumption and playback |
| [`src/speech_to_speech/pipeline/handler_types.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/pipeline/handler_types.py) | `TTSIn` and `TTSOut` type definitions |

## Summary

- **Pocket TTS voice cloning** works by extracting speaker embeddings from reference audio via `get_state_for_audio_prompt()` in [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py).

- **Three voice sources** are supported: built-in presets, local audio files, and Hugging Face repositories prefixed with `hf://`.

- **Streaming architecture** yields 16 kHz `int16` blocks suitable for real-time playback or network transmission.

- **CLI usage** requires `--tts pocket` plus `--pocket_tts_voice` pointing to your chosen source.

- **Programmatic usage** involves instantiating `PocketTTSHandler` with a `should_listen` Event and iterating over `process(TTSIn(text=...))`.

## Frequently Asked Questions

### What audio formats work for voice cloning reference files?

WAV and OGG formats are supported for local files. For Hugging Face repositories, the model expects standard audio file extensions. The `get_state_for_audio_prompt()` method handles format detection automatically according to the [`pocket_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/pocket_tts_handler.py) implementation.

### Can I clone multiple voices in the same pipeline instance?

Each `PocketTTSHandler` instance maintains a single voice state from initialization. To switch voices dynamically, create separate handler instances or re-instantiate with a new `voice` parameter. The voice state is cached at setup time, not per-generation.

### Why is my cloned voice output distorted or mismatched?

Check three common causes: insufficient or noisy reference audio (use 3-10 seconds of clean speech), mismatched `sample_rate` between handler and audio sink, or device overload (try reducing `blocksize` or switching from `cpu` to `cuda`). The handler outputs raw `int16` arrays that must not be resampled incorrectly downstream.

### How do I integrate Pocket TTS with custom audio outputs instead of LocalAudioStreamer?

Consume the generator returned by `handler.process(TTSIn(...))` directly. Each yielded value is a NumPy `int16` array. Route these to your custom sink—file writer, WebSocket, or hardware interface—without modification. The `WebSocketStreamer` in the repository demonstrates network transmission patterns you can adapt.