How to Use Custom Voice Files with Pocket TTS for Voice Cloning

To use custom voice files with Pocket TTS for voice cloning, pass a local audio file path, Hugging Face repository URL, or preset name to the pocket_tts_voice argument when configuring the PocketTTSHandler class.

The huggingface/speech-to-speech pipeline integrates the Kyutai Labs Pocket TTS model through a dedicated handler that supports voice cloning from arbitrary audio samples. By specifying a voice identifier in the handler arguments, you can generate real-time speech that mimics custom speakers while maintaining the pipeline's streaming architecture.

Understanding the Pocket TTS Architecture

The voice cloning capability is implemented across two primary modules that handle model initialization and voice state management.

Core Handler Implementation

The PocketTTSHandler class in src/speech_to_speech/TTS/pocket_tts_handler.py manages the text-to-speech conversion and voice embedding extraction. When the pipeline starts with module_kwargs.tts = "pocket", this handler instantiates the Kyutai Labs model and loads the specified voice state.

The handler automatically resamples the model's native 24 kHz output to your target sample rate (default 16 kHz) and streams audio in configurable blocks (default 512 samples) for real-time playback.

Configuration Arguments

Voice cloning parameters are defined in src/speech_to_speech/arguments_classes/pocket_tts_arguments.py through the PocketTTSHandlerArguments dataclass. The critical field for custom voices is pocket_tts_voice, which accepts three distinct input types and passes them to the model's get_state_for_audio_prompt method.

Supported Voice Input Formats

The handler accepts three voice identifier formats through the voice parameter:

• Preset name — Built-in speakers like "alba" or "jean" (default). These ship with the model and require no external files.
• Local audio file — Absolute path to a WAV file (e.g., "/path/to/my_voice.wav"). The handler extracts a voice embedding directly from the audio.
• Hugging Face repository — Repository path using the hf:// protocol (e.g., "hf://kyutai/tts-voices/custom_voice"). The handler downloads and caches the voice embedding from the specified repo.

Command-Line Usage Examples

You can specify custom voices directly via the CLI when launching the speech-to-speech pipeline:

# Use a built-in preset voice

python -m speech_to_speech.main --tts pocket \
    --pocket_tts_voice alba

# Clone from a local audio file

python -m speech_to_speech.main --tts pocket \
    --pocket_tts_voice /home/user/my_voice.wav

# Load a voice from a Hugging Face repository

python -m speech_to_speech.main --tts pocket \
    --pocket_tts_voice "hf://kyutai/tts-voices/custom_voice"

Programmatic Pipeline Integration

For custom implementations, instantiate the handler manually through the pipeline factory in src/speech_to_speech/s2s_pipeline.py:

from speech_to_speech.arguments_classes.pocket_tts_arguments import PocketTTSHandlerArguments
from speech_to_speech.s2s_pipeline import get_tts_handler
from queue import Queue
from threading import Event

# Configure custom voice arguments

pocket_args = PocketTTSHandlerArguments(
    pocket_tts_device="cuda",              # or "cpu", "mps"

    pocket_tts_voice="/path/to/custom_voice.wav",
    pocket_tts_sample_rate=16000,
    pocket_tts_blocksize=512,
    pocket_tts_max_tokens=80,
)

# Initialize pipeline queues and events

stop_event = Event()
lm_response_q = Queue()      # Receives TTSIn items from LLM

audio_out_q = Queue()        # Receives TTSOut chunks for playback

should_listen = Event()      # Signals STT to resume listening

# Build the handler

tts_handler = get_tts_handler(
    module_kwargs=type("M", (), {"tts": "pocket"}),
    stop_event=stop_event,
    lm_response_queue=lm_response_q,
    send_audio_chunks_queue=audio_out_q,
    should_listen=should_listen,
    chat_tts_handler_kwargs=None,
    facebook_mms_tts_handler_kwargs=None,
    pocket_tts_handler_kwargs=pocket_args,
    kokoro_tts_handler_kwargs=None,
    qwen3_tts_handler_kwargs=None,
)

Once initialized, calling tts_handler.process() streams synthesized audio using your custom voice embedding.

How Voice Loading Works Internally

Inside src/speech_to_speech/TTS/pocket_tts_handler.py, the setup() method processes the voice identifier through the following sequence:

# PocketTTSHandler.setup(...)

logger.info(f"Loading voice: {voice}")
self.voice_state = self.model.get_state_for_audio_prompt(voice)

The get_state_for_audio_prompt method (provided by the underlying pocket_tts library) abstracts the loading logic for all three voice source types. Whether you provide a preset string, local file path, or Hugging Face URL, the method returns a compatible voice state that the handler uses for subsequent audio generation.

Summary

• Three input formats are supported for Pocket TTS voice cloning: preset names, local WAV files, and Hugging Face repository paths via the hf:// protocol.
• Configuration occurs through PocketTTSHandlerArguments in src/speech_to_speech/arguments_classes/pocket_tts_arguments.py, specifically the pocket_tts_voice field.
• Automatic processing happens in src/speech_to_speech/TTS/pocket_tts_handler.py through get_state_for_audio_prompt, which handles embedding extraction and voice state initialization.
• Real-time streaming is maintained with automatic resampling from 24 kHz to your target rate and configurable block sizes for low-latency audio playback.

Frequently Asked Questions

What audio format should custom voice files use?

The handler accepts standard WAV files for local voice cloning. The pocket_tts library internally handles resampling and preprocessing, though using high-quality recordings (clean speech, minimal background noise) produces better cloning results. The specific sample rate of the input file is handled automatically by the model's embedding extractor.

Can I use GPU acceleration with Pocket TTS voice cloning?

Yes. Set pocket_tts_device="cuda" (or "mps" for Apple Silicon) in your PocketTTSHandlerArguments. The handler moves the model to the specified device during initialization in setup(), while voice state extraction via get_state_for_audio_prompt also respects the device configuration for accelerated processing.

How does the handler handle different sample rates?

The Pocket TTS model generates audio at 24 kHz natively. The handler in src/speech_to_speech/TTS/pocket_tts_handler.py automatically resamples output to the sample_rate specified in your arguments (default 16 kHz) before streaming chunks to the audio output queue. This ensures compatibility with the pipeline's audio streamer regardless of the model's native rate.

Where are the preset voices defined?

Preset voices like "jean" (the default) and "alba" are built into the Kyutai Labs model weights and referenced by the handler's docstring in src/speech_to_speech/TTS/pocket_tts_handler.py. These require no external files or downloads; the get_state_for_audio_prompt method retrieves their embeddings directly from the model checkpoint.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}