How to Configure and Use Different Text‑to‑Speech (TTS) Backends in Hugging Face Speech‑to‑Speech

You configure different TTS backends in the speech‑to‑speech repository by setting the qwen3_tts_backend argument to "mlx", "ggml", or "torch" (or letting it auto‑select based on your platform), then passing the corresponding arguments via Qwen3TTSHandlerArguments when instantiating the handler.

The Hugging Face speech‑to‑speech repository abstracts every TTS engine behind a common handler interface, making it straightforward to switch between MLX (Apple Silicon), GGML, and PyTorch backends without changing your pipeline code. This guide walks through the architecture, configuration options, and runnable examples for configuring TTS backends in both CLI and Python workflows.

---

Understanding the TTS Backend Architecture

Every TTS implementation in the repository inherits from BaseHandler[TTSIn, TTSOut] defined in src/speech_to_speech/baseHandler.py. This base class standardizes the setup(), process(), and cleanup() lifecycle that s2s_pipeline.py orchestrates.

The typed queues TTSIn and TTSOut (from src/speech_to_speech/pipeline/handler_types.py) carry text, language, and runtime configuration into the handler and stream raw audio chunks back out.

For Qwen‑3‑TTS—the most feature‑rich handler—the backend selection happens automatically at runtime based on platform detection, with optional user overrides.

---

Backend Selection Logic in Qwen3TTSHandler

In src/speech_to_speech/TTS/qwen3_tts_handler.py, the setup() method determines which backend to load:

self.backend = "mlx" if platform == "darwin" else "faster_qwen3_tts"  # line 51

The three available backends are:

Backend	Platform	Implementation	Selection Method
MLX	Apple Silicon only	mlx_audio.tts.utils.load_model	Auto‑selected when platform == "darwin"
GGML	Linux, macOS, Windows	FasterQwen3TTS with GGUF weights	--qwen3_tts_backend ggml (default)
Torch	Any (CUDA/CPU)	FasterQwen3TTS PyTorch	--qwen3_tts_backend torch

The faster‑qwen3‑tts backends are normalized by _normalize_faster_backend() (lines 66‑72), which converts user input into the correct internal format.

---

Configuring TTS Backends via Command Line

Use the --qwen3_tts_backend flag combined with backend‑specific options:

# Default GGML backend with BF16 quantization (Linux/macOS)

speech-to-speech \
  --tts qwen3 \
  --qwen3_tts_backend ggml \
  --qwen3_tts_ggml_quantization BF16

# Torch backend on CUDA

speech-to-speech \
  --tts qwen3 \
  --qwen3_tts_backend torch \
  --qwen3_tts_device cuda \
  --qwen3_tts_dtype bfloat16

# MLX backend (auto‑selected on Apple Silicon, no flags needed)

speech-to-speech --tts qwen3

Other relevant flags include:

• --qwen3_tts_streaming_chunk_size — controls latency vs. throughput (default: 4 for MLX, 8 for faster backends)
• --qwen3_tts_model_name — specify a custom model ID or local path
• --qwen3_tts_device — cpu, cuda, or mps

---

Configuring TTS Backends Programmatically

Import Qwen3TTSHandlerArguments and Qwen3TTSHandler to build configurations in Python:

from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSHandlerArguments
from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

# Configure Torch backend with custom model

tts_args = Qwen3TTSHandlerArguments(
    qwen3_tts_backend="torch",
    qwen3_tts_device="cuda",
    qwen3_tts_model_name="my-org/custom-qwen3-tts",
    qwen3_tts_dtype="float16",
)

handler = Qwen3TTSHandler(**tts_args.__dict__)
pipeline = SpeechToSpeechPipeline(tts_handler=handler)
pipeline.run()

All fields from the arguments dataclass are passed verbatim to the handler constructor, which then initializes the appropriate backend during setup().

---

Backend‑Specific Initialization Details

MLX Backend (Apple Silicon)

Lines 39‑44 in qwen3_tts_handler.py load the model via:

from mlx_audio.tts.utils import load_model
self.model = load_model(DEFAULT_MLX_MODEL)

The default model is the MLX‑converted Qwen‑3‑TTS variant. Streaming uses a chunk size of 4 tokens by default.

GGML and Torch Backends

For faster‑qwen3‑tts, the handler imports FasterQwen3TTS and calls from_pretrained() with device, dtype, and GGML‑specific options (lines 201‑228). The _model_type() method (lines 62‑68) inspects model metadata to route requests to the correct generation method:

• Voice‑clone: _process_voice_clone (lines 71‑89)
• Custom‑voice: _process_custom_voice (lines 709‑739)
• Voice‑design: _process_voice_design

---

Complete Configuration Examples

Example 1: GGML Backend with Default Settings

from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSHandlerArguments
from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler
from speech_to_speech.s2s_pipeline import SpeechToSpeechPipeline

args = Qwen3TTSHandlerArguments()  # defaults: ggml backend, BF16 quantization

tts_handler = Qwen3TTSHandler(**args.__dict__)

pipeline = SpeechToSpeechPipeline(tts_handler=tts_handler)
pipeline.run()

Example 2: Torch Backend with Explicit GPU Settings

from speech_to_speech.arguments_classes.qwen3_tts_arguments import Qwen3TTSHandlerArguments
from speech_to_speech.TTS.qwen3_tts_handler import Qwen3TTSHandler

args = Qwen3TTSHandlerArguments(
    qwen3_tts_backend="torch",
    qwen3_tts_device="cuda",
    qwen3_tts_dtype="bfloat16",
)

handler = Qwen3TTSHandler(**args.__dict__)

Example 3: CustomVoice Model Without Reference Audio

args = Qwen3TTSHandlerArguments(
    qwen3_tts_model_name="Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice",
    qwen3_tts_speaker="Aiden",
)

handler = Qwen3TTSHandler(**args.__dict__)

The handler automatically invokes _process_custom_voice for streaming generation.

Example 4: Voice‑Clone with Reference Audio

args = Qwen3TTSHandlerArguments(
    qwen3_tts_ref_audio="examples/reference.wav",
    qwen3_tts_ref_text="Hello, I am a cloned voice.",
)

handler = Qwen3TTSHandler(**args.__dict__)

This triggers _process_voice_clone, which normalizes the reference audio and streams via the selected backend.

---

Runtime Voice Overrides

The handler supports session‑level voice switching through _apply_session_voice_override() (lines 94‑138). If a realtime response contains a voice field, the handler rewrites speaker or ref_audio on‑the‑fly without restarting the backend. This works across all backends and generation modes.

---

Alternative TTS Backends

The same handler pattern applies to other TTS implementations in the repository:

Handler	File	Use Case
Pocket TTS	src/speech_to_speech/TTS/pocket_tts_handler.py	Lightweight, edge‑deployable TTS
Kokoro	src/speech_to_speech/TTS/kokoro_handler.py	Distinctive voice characteristics
Facebook MMS	src/speech_to_speech/TTS/facebookmms_handler.py	Multilingual TTS with GGUF support
ChatTTS	src/speech_to_speech/TTS/chatTTS_handler.py	Remote API‑based commercial TTS

Each follows the BaseHandler contract with matching *HandlerArguments dataclasses in src/speech_to_speech/arguments_classes/.

---

Summary

• Auto‑selection: MLX on Apple Silicon, GGML elsewhere—no configuration required
• Explicit override: Set qwen3_tts_backend to "ggml" or "torch" for faster‑qwen3‑tts
• Configuration layer: Qwen3TTSHandlerArguments maps CLI flags and Python parameters to handler initialization
• Plug‑and‑play architecture: All TTS backends implement BaseHandler, enabling seamless swaps without pipeline changes
• Runtime flexibility: Voice overrides apply across all backends without restart

---

Frequently Asked Questions

How do I force MLX on a non‑Apple platform or disable it on Apple Silicon?

MLX is hardcoded to Apple Silicon detection in line 51 of qwen3_tts_handler.py. To override, you would need to modify the source or use a different handler. There is no runtime flag to force MLX on non‑Darwin platforms—use the Torch or GGML backends instead.

What quantization options work with the GGML backend?

The qwen3_tts_ggml_quantization argument accepts standard GGUF quantizations including BF16 (default), Q8_0, Q6_K, Q5_K_M, Q4_K_M, and others supported by the underlying GGML runtime. Lower quantization reduces memory at the cost of voice quality.

Can I switch TTS backends without restarting the pipeline?

Not at the handler level—setup() initializes backend‑specific resources. However, you can instantiate multiple SpeechToSpeechPipeline objects with different handlers and route between them at the application layer. Session voice overrides (changing speaker or reference audio) work within a running handler.

Does the Torch backend require CUDA?

No. The Torch backend falls back to CPU execution when CUDA is unavailable. Set qwen3_tts_device="cpu" explicitly, or the handler will auto‑detect. Performance on CPU is significantly slower than GGML for the same model size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}