How to Use Different Vocoders with Speech-to-Speech Models in Hugging Face's speech-to-speech Library

You can swap vocoders in the speech-to-speech pipeline by configuring the Qwen3TTSHandlerArguments with your desired backend (ggml or torch), quantization level, and optional custom GGUF files for the talker and codec components.

The Hugging Face speech-to-speech repository decouples the neural model from the vocoder, letting you experiment with different audio synthesis backends. This guide shows you how to use different vocoders with speech-to-speech models by leveraging the configuration options in Qwen3TTSHandlerArguments.

Architecture: Where Vocoder Configuration Lives

The pipeline separates concerns across four layers. Understanding this structure helps you locate the right files when customizing your setup.

Layer	Responsibility	Source File
Pipeline entry point	Parses CLI/JSON arguments and constructs the handler chain	src/speech_to_speech/s2s_pipeline.py
Argument dataclass	Holds user-exposed vocoder options	src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py
TTS handler	Instantiates model and vocoder, streams audio	src/speech_to_speech/TTS/qwen3_tts_handler.py
Backend implementations	Execute actual vocoder inference	mlx-audio (Apple Silicon) or faster-qwen3-tts (CUDA/CPU)

When you specify --tts qwen3, the factory function get_tts_handler in s2s_pipeline.py (lines 389–411) creates a Qwen3TTSHandler. The handler's setup() method then validates and applies your vocoder selections.

Vocoder Configuration Options

The following arguments in Qwen3TTSHandlerArguments control which vocoder loads and how it runs:

Argument	Purpose	Common Values
qwen3_tts_backend	Selects the inference library	"ggml" (default), "torch"
qwen3_tts_ggml_quantization	Quantization for GGML backend	"BF16", "Q8_0", "Q4_K_M", "F32"
qwen3_tts_gguf_talker_path	Path to custom speaker GGUF file	Local .gguf file path
qwen3_tts_gguf_codec_path	Path to custom vocoder/codec GGUF file	Local .gguf file path
qwen3_tts_mlx_quantization	Quantization for MLX (Apple Silicon)	"bf16", "4bit", "6bit", "8bit"
qwen3_tts_ref_audio / qwen3_tts_ref_spk / qwen3_tts_ref_rvq	Voice cloning references	Paths to .wav, .spk, or .rvq files

Backend-Specific Vocoder Behavior

MLX backend (platform == "darwin"): The vocoder is integrated into the MLX model loaded via mlx_audio.tts.utils.load_model. You adjust precision with qwen3_tts_mlx_quantization, but cannot swap the vocoder independently.

GGML/Torch backend (platform != "darwin"): The handler instantiates FasterQwen3TTS, passing your selected qwen3_tts_backend, qwen3_tts_ggml_quantization, and optional GGUF paths. This is where you inject entirely different vocoders via custom codec and talker files.

Method 1: Configure Vocoders via Command Line

The fastest way to use different vocoders with speech-to-speech models is CLI flags. This example forces the Torch backend with custom GGUF files:

python -m speech_to_speech.main \
    --tts qwen3 \
    --qwen3_tts_backend torch \
    --qwen3_tts_gguf_talker_path /path/to/custom_talker.gguf \
    --qwen3_tts_gguf_codec_path /path/to/custom_codec.gguf \
    --qwen3_tts_ggml_quantization Q8_0

The _validate_ggml_options() method in qwen3_tts_handler.py checks that your GGUF paths exist and are compatible with your selected backend before loading.

Method 2: Configure Vocoders Programmatically

For integration into larger applications, build the Qwen3TTSHandlerArguments dataclass directly:

from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSHandlerArguments
from speech_to_speech.s2s_pipeline import ParsedArguments, prepare_all_args

# Configure your custom vocoder

qwen_args = Qwen3TTSHandlerArguments(
    qwen3_tts_backend="torch",
    qwen3_tts_gguf_talker_path="/data/my_talker.gguf",
    qwen3_tts_gguf_codec_path="/data/my_codec.gguf",
    qwen3_tts_ggml_quantization="Q8_0",
)

# Assemble complete argument suite

parsed = ParsedArguments(
    module_kwargs=module_args,
    socket_receiver_kwargs=socket_recv_args,
    socket_sender_kwargs=socket_send_args,
    websocket_streamer_kwargs=ws_stream_args,
    vad_handler_kwargs=vad_args,
    whisper_stt_handler_kwargs=whisper_args,
    language_model_handler_kwargs=lm_args,
    responses_api_language_model_handler_kwargs=api_lm_args,
    chat_tts_handler_kwargs=chat_tts_args,
    facebook_mms_tts_handler_kwargs=fb_mms_args,
    pocket_tts_handler_kwargs=pocket_args,
    kokoro_tts_handler_kwargs=kokoro_args,
    qwen3_tts_handler_kwargs=qwen_args,  # Your vocoder configuration

)

# Apply device mapping and final preparation

prepare_all_args(**vars(parsed))

Voice Cloning with Custom Vocoders

The speech-to-speech pipeline supports voice cloning through reference audio or pre-computed speaker embeddings. To use a different vocoder for cloned voices:

• Point qwen3_tts_ref_audio to a .wav file for on-the-fly embedding extraction
• Use qwen3_tts_ref_spk (.spk file) or qwen3_tts_ref_rvq (.rvq file) for pre-computed embeddings
• Change qwen3_tts_speaker for CustomVoice speakers

These references flow through the same codec specified in qwen3_tts_gguf_codec_path, letting you combine voice cloning with custom vocoder architectures.

Key Source Files Reference

File	Description
src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py	Dataclass defining all vocoder-related CLI arguments
src/speech_to_speech/TTS/qwen3_tts_handler.py	Handler logic for backend selection, validation, and audio streaming
src/speech_to_speech/s2s_pipeline.py	Pipeline factory (get_tts_handler) that instantiates handlers
src/speech_to_speech/utils/mlx_lock.py	Serialization utilities for MLX inference on Apple Silicon
src/speech_to_speech/arguments_classes/module_arguments.py	Module-level flags including --tts selector

Summary

• Separate concerns: The speech-to-speech pipeline splits model inference from vocoder synthesis, enabling flexible experimentation.
• Two backends: Use mlx-audio on Apple Silicon (vocoder bundled) or faster-qwen3-tts on CUDA/CPU (swappable GGUF vocoders).
• Swap via GGUF paths: On non-Apple platforms, set qwen3_tts_gguf_codec_path and qwen3_tts_gguf_talker_path to load entirely different vocoder weights.
• Quantization control: Fine-tune speed/quality tradeoffs with qwen3_tts_ggml_quantization or qwen3_tts_mlx_quantization.
• Entry points: Configure through CLI flags for quick tests or Qwen3TTSHandlerArguments for programmatic integration.

Frequently Asked Questions

Can I use a custom vocoder on Apple Silicon?

No. The MLX backend in src/speech_to_speech/TTS/qwen3_tts_handler.py loads models through mlx_audio.tts.utils.load_model, which bundles the vocoder. You can only adjust quantization precision via qwen3_tts_mlx_quantization, not swap the vocoder architecture. For full vocoder customization, use the GGML or Torch backend on Linux/Windows.

What file format do custom vocoders need?

Custom vocoders and talkers use GGUF format (a binary format for GGML models). Specify paths via qwen3_tts_gguf_codec_path (vocoder weights) and qwen3_tts_gguf_talker_path (speaker/model weights). The handler validates these paths in _validate_ggml_options() before loading.

How do I optimize vocoder performance for real-time streaming?

Three levers control speed: quantization (Q4_K_M for fastest, BF16 for quality), backend selection (ggml for CPU efficiency, torch for CUDA optimization), and platform (MLX on Apple Silicon has lower overhead). For latency-critical applications, start with Q8_0 quantization and profile on your target hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}