# How to Use Different Vocoders with Speech-to-Speech Models in Hugging Face's speech-to-speech Library

> Learn to swap vocoders in Hugging Face speech-to-speech models. Configure Qwen3TTSHandlerArguments with ggml or torch backends for custom talker and codec components.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-02

---

**You can swap vocoders in the speech-to-speech pipeline by configuring the `Qwen3TTSHandlerArguments` with your desired backend (`ggml` or `torch`), quantization level, and optional custom GGUF files for the talker and codec components.**

The Hugging Face **speech-to-speech** repository decouples the neural model from the vocoder, letting you experiment with different audio synthesis backends. This guide shows you how to use different vocoders with speech-to-speech models by leveraging the configuration options in `Qwen3TTSHandlerArguments`.

## Architecture: Where Vocoder Configuration Lives

The pipeline separates concerns across four layers. Understanding this structure helps you locate the right files when customizing your setup.

| Layer | Responsibility | Source File |
|-------|---------------|-------------|
| Pipeline entry point | Parses CLI/JSON arguments and constructs the handler chain | [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) |
| Argument dataclass | Holds user-exposed vocoder options | [`src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py) |
| TTS handler | Instantiates model and vocoder, streams audio | [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) |
| Backend implementations | Execute actual vocoder inference | `mlx-audio` (Apple Silicon) or `faster-qwen3-tts` (CUDA/CPU) |

When you specify `--tts qwen3`, the factory function `get_tts_handler` in [`s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/s2s_pipeline.py) (lines 389–411) creates a `Qwen3TTSHandler`. The handler's `setup()` method then validates and applies your vocoder selections.

## Vocoder Configuration Options

The following arguments in `Qwen3TTSHandlerArguments` control which vocoder loads and how it runs:

| Argument | Purpose | Common Values |
|----------|---------|---------------|
| `qwen3_tts_backend` | Selects the inference library | `"ggml"` (default), `"torch"` |
| `qwen3_tts_ggml_quantization` | Quantization for GGML backend | `"BF16"`, `"Q8_0"`, `"Q4_K_M"`, `"F32"` |
| `qwen3_tts_gguf_talker_path` | Path to custom speaker GGUF file | Local `.gguf` file path |
| `qwen3_tts_gguf_codec_path` | Path to custom vocoder/codec GGUF file | Local `.gguf` file path |
| `qwen3_tts_mlx_quantization` | Quantization for MLX (Apple Silicon) | `"bf16"`, `"4bit"`, `"6bit"`, `"8bit"` |
| `qwen3_tts_ref_audio` / `qwen3_tts_ref_spk` / `qwen3_tts_ref_rvq` | Voice cloning references | Paths to `.wav`, `.spk`, or `.rvq` files |

### Backend-Specific Vocoder Behavior

**MLX backend** (`platform == "darwin"`): The vocoder is integrated into the MLX model loaded via `mlx_audio.tts.utils.load_model`. You adjust precision with `qwen3_tts_mlx_quantization`, but cannot swap the vocoder independently.

**GGML/Torch backend** (`platform != "darwin"`): The handler instantiates `FasterQwen3TTS`, passing your selected `qwen3_tts_backend`, `qwen3_tts_ggml_quantization`, and optional GGUF paths. This is where you inject entirely different vocoders via custom `codec` and `talker` files.

## Method 1: Configure Vocoders via Command Line

The fastest way to use different vocoders with speech-to-speech models is CLI flags. This example forces the Torch backend with custom GGUF files:

```bash
python -m speech_to_speech.main \
    --tts qwen3 \
    --qwen3_tts_backend torch \
    --qwen3_tts_gguf_talker_path /path/to/custom_talker.gguf \
    --qwen3_tts_gguf_codec_path /path/to/custom_codec.gguf \
    --qwen3_tts_ggml_quantization Q8_0

```

The `_validate_ggml_options()` method in [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py) checks that your GGUF paths exist and are compatible with your selected backend before loading.

## Method 2: Configure Vocoders Programmatically

For integration into larger applications, build the `Qwen3TTSHandlerArguments` dataclass directly:

```python
from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSHandlerArguments
from speech_to_speech.s2s_pipeline import ParsedArguments, prepare_all_args

# Configure your custom vocoder

qwen_args = Qwen3TTSHandlerArguments(
    qwen3_tts_backend="torch",
    qwen3_tts_gguf_talker_path="/data/my_talker.gguf",
    qwen3_tts_gguf_codec_path="/data/my_codec.gguf",
    qwen3_tts_ggml_quantization="Q8_0",
)

# Assemble complete argument suite

parsed = ParsedArguments(
    module_kwargs=module_args,
    socket_receiver_kwargs=socket_recv_args,
    socket_sender_kwargs=socket_send_args,
    websocket_streamer_kwargs=ws_stream_args,
    vad_handler_kwargs=vad_args,
    whisper_stt_handler_kwargs=whisper_args,
    language_model_handler_kwargs=lm_args,
    responses_api_language_model_handler_kwargs=api_lm_args,
    chat_tts_handler_kwargs=chat_tts_args,
    facebook_mms_tts_handler_kwargs=fb_mms_args,
    pocket_tts_handler_kwargs=pocket_args,
    kokoro_tts_handler_kwargs=kokoro_args,
    qwen3_tts_handler_kwargs=qwen_args,  # Your vocoder configuration

)

# Apply device mapping and final preparation

prepare_all_args(**vars(parsed))

```

## Voice Cloning with Custom Vocoders

The speech-to-speech pipeline supports voice cloning through reference audio or pre-computed speaker embeddings. To use a different vocoder for cloned voices:

- Point `qwen3_tts_ref_audio` to a `.wav` file for on-the-fly embedding extraction
- Use `qwen3_tts_ref_spk` (`.spk` file) or `qwen3_tts_ref_rvq` (`.rvq` file) for pre-computed embeddings
- Change `qwen3_tts_speaker` for CustomVoice speakers

These references flow through the same codec specified in `qwen3_tts_gguf_codec_path`, letting you combine voice cloning with custom vocoder architectures.

## Key Source Files Reference

| File | Description |
|------|-------------|
| [`src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/qwen3_tts_arguments.py) | Dataclass defining all vocoder-related CLI arguments |
| [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) | Handler logic for backend selection, validation, and audio streaming |
| [`src/speech_to_speech/s2s_pipeline.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/s2s_pipeline.py) | Pipeline factory (`get_tts_handler`) that instantiates handlers |
| [`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) | Serialization utilities for MLX inference on Apple Silicon |
| [`src/speech_to_speech/arguments_classes/module_arguments.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/arguments_classes/module_arguments.py) | Module-level flags including `--tts` selector |

## Summary

- **Separate concerns**: The speech-to-speech pipeline splits model inference from vocoder synthesis, enabling flexible experimentation.
- **Two backends**: Use `mlx-audio` on Apple Silicon (vocoder bundled) or `faster-qwen3-tts` on CUDA/CPU (swappable GGUF vocoders).
- **Swap via GGUF paths**: On non-Apple platforms, set `qwen3_tts_gguf_codec_path` and `qwen3_tts_gguf_talker_path` to load entirely different vocoder weights.
- **Quantization control**: Fine-tune speed/quality tradeoffs with `qwen3_tts_ggml_quantization` or `qwen3_tts_mlx_quantization`.
- **Entry points**: Configure through CLI flags for quick tests or `Qwen3TTSHandlerArguments` for programmatic integration.

## Frequently Asked Questions

### Can I use a custom vocoder on Apple Silicon?

No. The MLX backend in [`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) loads models through `mlx_audio.tts.utils.load_model`, which bundles the vocoder. You can only adjust quantization precision via `qwen3_tts_mlx_quantization`, not swap the vocoder architecture. For full vocoder customization, use the GGML or Torch backend on Linux/Windows.

### What file format do custom vocoders need?

Custom vocoders and talkers use **GGUF format** (a binary format for GGML models). Specify paths via `qwen3_tts_gguf_codec_path` (vocoder weights) and `qwen3_tts_gguf_talker_path` (speaker/model weights). The handler validates these paths in `_validate_ggml_options()` before loading.

### How do I optimize vocoder performance for real-time streaming?

Three levers control speed: **quantization** (`Q4_K_M` for fastest, `BF16` for quality), **backend selection** (`ggml` for CPU efficiency, `torch` for CUDA optimization), and **platform** (MLX on Apple Silicon has lower overhead). For latency-critical applications, start with `Q8_0` quantization and profile on your target hardware.