Which TTS Backends Support Voice Cloning in huggingface/speech-to-speech and How to Configure Them

Only Qwen 3 TTS and Pocket TTS support voice cloning in the huggingface/speech-to-speech library, configured via reference audio paths, speaker embeddings, or voice file arguments while Facebook MMS, Kokoro, and ChatTTS provide preset-only synthesis without cloning capabilities.

The huggingface/speech-to-speech repository offers multiple Text-to-Speech (TTS) backend options, but voice cloning—which synthesizes speech mimicking a reference speaker—is limited to specific handlers. Understanding which TTS backends support voice cloning and how to configure them is essential for building personalized speech-to-speech pipelines.

TTS Backends That Support Voice Cloning

Qwen 3 TTS

Qwen 3 TTS provides the most flexible voice cloning implementation in the library, supporting three distinct cloning modes through different input types.

Configuration parameters (defined in src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py):

• qwen3_tts_ref_audio – filesystem path to a reference WAV or MP3 file
• qwen3_tts_ref_spk – path to a pre-computed speaker embedding file (.spk extension)
• qwen3_tts_ref_rvq – path to pre-computed acoustic codes (.rvq extension)
• qwen3_tts_xvec_only – boolean flag enabling x-vector-only cloning mode

The handler implementation in src/speech_to_speech/TTS/qwen3_tts_handler.py parses these arguments in its setup method and internally branches to the appropriate cloning pathway. The x-vector-only mode is specifically recommended for clean starts and language switching scenarios.

Pocket TTS

Pocket TTS offers cloning through its flexible voice parameter, which accepts both preset identifiers and custom audio sources.

Configuration via the voice argument (handled in src/speech_to_speech/TTS/pocket_tts_handler.py):

• Preset strings: "alba", "marius", or other built-in voice names
• Local audio files: filesystem paths like "my_voice.wav"
• Hugging Face repositories: URLs formatted as "hf://kyutai/tts-voices/custom_voice"

When voice points to an audio file or HF URL, the handler's setup method calls self.model.get_state_for_audio_prompt(voice) to extract a speaker state, which is then reused for all subsequent generations. This differs from Qwen 3 TTS by not requiring separate embedding pre-computation.

TTS Backends Without Voice Cloning

The following backends are available but do not expose cloning functionality:

Backend	Cloning Support	Implementation File
Facebook MMS TTS	❌ No—language-specific VITS synthesis only	src/speech_to_speech/TTS/facebookmms_handler.py
Kokoro TTS	❌ No—fixed speaker presets only	src/speech_to_speech/TTS/kokoro_handler.py
ChatTTS	❌ No—fixed speaker presets only	src/speech_to_speech/TTS/chatTTS_handler.py

These handlers generate speech solely from text input using their respective models without reference speaker conditioning.

Configuration Flow Through the Pipeline

All TTS backend selection and parameter passing flows through src/speech_to_speech/s2s_pipeline.py. The pipeline instantiates handlers based on RuntimeConfig, which aggregates arguments from src/speech_to_speech/cli.py or direct programmatic configuration.

Key stages:

1. CLI parsing – cli.py captures user arguments including TTS handler selection and cloning-specific flags
2. Runtime configuration – RuntimeConfig dataclass stores validated parameters
3. Handler instantiation – Pipeline creates the selected handler and calls its setup method with configuration arguments
4. Resource storage – Cloning-capable handlers persist reference resources (self.ref_audio, self.ref_spk, self.voice_state) for use in process()

Practical Configuration Examples

Qwen 3 TTS with Raw Reference Audio

from speech_to_speech.pipeline import SpeechToSpeechPipeline
from speech_to_speech.runtime_config import RuntimeConfig

cfg = RuntimeConfig(
    tts_handler="qwen3",
    qwen3_tts_ref_audio="samples/reference.wav",
    qwen3_tts_ref_text="Hello, this is my voice.",
    qwen3_tts_xvec_only=True,  # Recommended for cross-language cloning

)

pipeline = SpeechToSpeechPipeline(cfg)
pipeline.run(...)

Qwen 3 TTS with Pre-computed Speaker Embedding

cfg = RuntimeConfig(
    tts_handler="qwen3",
    qwen3_tts_ref_spk="embeddings/speaker.spk",  # Pre-extracted embedding

    qwen3_tts_xvec_only=True,
)

Pocket TTS with Local Voice File

cfg = RuntimeConfig(
    tts_handler="pocket",
    pocket_tts_voice="recordings/my_voice.wav",
    pocket_tts_device="cuda",
)

Pocket TTS with Hugging Face Voice Repository

cfg = RuntimeConfig(
    tts_handler="pocket",
    pocket_tts_voice="hf://kyutai/tts-voices/custom_speaker",
)

Command-Line Usage

The CLI mirrors programmatic configuration:

Qwen 3 TTS cloning via CLI:

python -m speech_to_speech.cli \
    --tts-handler qwen3 \
    --qwen3-tts-ref-audio samples/reference.wav \
    --qwen3-tts-ref-text "This is the reference transcription." \
    --qwen3-tts-xvec-only

Pocket TTS cloning via CLI:

python -m speech_to_speech.cli \
    --tts-handler pocket \
    --pocket-tts-voice my_voice.wav

Summary

• Qwen 3 TTS offers the most versatile voice cloning with three input modes (raw audio, speaker embeddings, acoustic codes) plus x-vector-only optimization for language switching
• Pocket TTS provides streamlined cloning through its unified voice parameter accepting presets, local files, or HF repository URLs
• Facebook MMS, Kokoro, and ChatTTS do not support voice cloning and are limited to preset-based synthesis
• All cloning configuration flows through RuntimeConfig into handler setup methods, with resources stored for reuse during process() calls
• Source implementations reside in src/speech_to_speech/TTS/qwen3_tts_handler.py and src/speech_to_speech/TTS/pocket_tts_handler.py

Frequently Asked Questions

What is the difference between Qwen 3 TTS and Pocket TTS voice cloning approaches?

Qwen 3 TTS exposes explicit cloning modes with separate parameters for different input types (raw audio, embeddings, acoustic codes) and includes the x-vector-only optimization flag. Pocket TTS uses a unified voice parameter that auto-detects whether you've provided a preset name, local file path, or HF URL, making it simpler but less configurable for advanced use cases.

Can I use voice cloning with Facebook MMS TTS or other backends?

No. According to the source code in src/speech_to_speech/TTS/facebookmms_handler.py, Facebook MMS TTS only synthesizes from text using language-specific VITS models. Similarly, src/speech_to_speech/TTS/kokoro_handler.py and src/speech_to_speech/TTS/chatTTS_handler.py implement fixed-preset synthesis without speaker conditioning. Only Qwen 3 TTS and Pocket TTS currently support cloning.

What is the x-vector-only mode in Qwen 3 TTS and when should I use it?

The qwen3_tts_xvec_only boolean flag enables a cloning mode that uses only speaker x-vectors rather than full acoustic conditioning. According to the implementation, this mode is specifically recommended for clean generation starts and when switching between languages, as it reduces interference from reference audio content while preserving speaker identity characteristics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}