How to Benchmark TTS Quantization Variants on macOS: Complete Guide

Run python scripts/benchmark_tts.py --qwen3_mlx_quantizations bf16 4bit 6bit 8bit on macOS to compare Qwen-3-TTS quantization levels by latency, RTF, and warm-up time.

The huggingface/speech-to-speech repository ships with purpose-built tooling to evaluate Text-to-Speech (TTS) performance across quantization configurations. On Apple Silicon Macs, you can benchmark the MLX-optimized Qwen-3-TTS model at bf16, 4bit, 6bit, and 8bit precisions using a single command-line invocation. This guide walks through the architecture, exact commands, and result interpretation.

---

Architecture of the Benchmarking System

Three core components coordinate TTS quantization benchmarking on macOS:

Component	Role	Source Location
benchmark_tts.py	CLI entry point, target builder, and metrics aggregator	[scripts/benchmark_tts.py](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py)
Qwen3TTSHandler	Platform detection, MLX model resolution, and quantization suffix application	[src/speech_to_speech/TTS/qwen3_tts_handler.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py)
mlx_lock.py	Serializes access to the MLX runtime across concurrent handler instances	[src/speech_to_speech/utils/mlx_lock.py](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py)

Key Functions and Constants

• VALID_QWEN3_MLX_QUANTIZATIONS — Hardcoded tuple at line 31 of benchmark_tts.py defining permitted values: ("4bit", "6bit", "8bit", "bf16")
• normalize_qwen3_mlx_quantizations — Validates user input against the allowed set (lines 12-30)
• build_benchmark_targets — Expands qwen3 into enumerated targets like qwen3[4bit] with per-quantization handler_kwargs (lines 34-51)
• _apply_mlx_quantization_suffix — Inserts or replaces quantization suffixes in MLX model identifiers (lines 42-52 of qwen3_tts_handler.py)
• _resolve_mlx_model_name — Applies default 6-bit quantization when unspecified (lines 64-78)

---

Running the Benchmark on macOS

Prerequisites

# Clone repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

# Install MLX dependencies

pip install mlx-audio mlx-community

Benchmark Command

python scripts/benchmark_tts.py \
  --text "Benchmarking Qwen-3-TTS quantization variants on Apple Silicon" \
  --handlers qwen3 \
  --iterations 10 \
  --qwen3_mlx_quantizations bf16 4bit 6bit 8bit \
  --output macos_quantization_bench.json

Parameter breakdown:

• --handlers qwen3 — Selects the Qwen-3-TTS backend (auto-detects MLX on darwin)
• --qwen3_mlx_quantizations — Space-separated list from VALID_QWEN3_MLX_QUANTIZATIONS
• --iterations 10 — Number of synthesis passes per quantization level (higher = more stable statistics)
• --output — Destination for structured JSON results

Execution Flow

1. CLI parsing — argparse populates args.qwen3_mlx_quantizations with your quantization list
2. Target expansion — build_benchmark_targets generates four distinct benchmark targets:
   • qwen3[bf16] → handler_kwargs: {"mlx_quantization": "bf16"}
   • qwen3[4bit] → handler_kwargs: {"mlx_quantization": "4bit"}
   • qwen3[6bit] → handler_kwargs: {"mlx_quantization": "6bit"}
   • qwen3[8bit] → handler_kwargs: {"mlx_quantization": "8bit"}
3. Handler instantiation — For each target, benchmark_handler creates a Qwen3TTSHandler; setup() detects platform == "darwin" and routes to MLX
4. Model resolution — _resolve_mlx_model_name calls _apply_mlx_quantization_suffix to transform base model names (e.g., mlx-community/Qwen3-TTS → mlx-community/Qwen3-TTS-4bit)
5. Streaming benchmark — The loop at lines 165-200 measures:
   • Warm-up time: Model loading and weight decompression
   • Time-to-first-chunk: Latency before audio generation begins
   • Total inference time: End-to-end synthesis duration
   • Real-time factor (RTF): Audio duration divided by inference time

---

Understanding the Results

The JSON output follows this schema:

{
  "timestamp": "2026-08-07 14:23:11",
  "results": [
    {
      "handler": "qwen3[bf16]",
      "warmup_time": 1.247,
      "avg_inference_time": 2.184,
      "min_inference_time": 2.091,
      "max_inference_time": 2.312,
      "std_inference_time": 0.076,
      "avg_audio_duration": 4.52,
      "avg_rtf": 2.07,
      "total_iterations": 10,
      "errors": []
    },
    {
      "handler": "qwen3[4bit]",
      "warmup_time": 0.823,
      "avg_inference_time": 1.156,
      "avg_rtf": 3.91,
      ...
    }
  ]
}

Key Metrics for Quantization Comparison

Metric	Interpretation	Optimization Target
warmup_time	Model loading overhead	Lower = faster cold-start
avg_inference_time	Mean synthesis latency	Lower = more responsive
avg_rtf	Throughput multiplier	Higher = better efficiency; RTF > 1 means faster-than-real-time

Quantization Trade-offs

• bf16: Highest precision, largest memory footprint, slowest inference
• 8bit: Modest compression with minimal quality degradation
• 6bit: Balanced default (used when no quantization specified)
• 4bit: Maximum speed and smallest size, potential quality reduction

Compare avg_rtf across entries: qwen3[4bit] typically achieves ~1.5-2× higher RTF than qwen3[bf16], representing substantial efficiency gains for production deployments.

---

macOS-Specific Considerations

Platform Enforcement

The quantization benchmarking path requires Apple Silicon with MLX. The detection logic in Qwen3TTSHandler.setup() (around line 90 of qwen3_tts_handler.py) branches:

if platform.system() == "darwin":
    self.backend = "mlx"
    self.model_name = self._resolve_mlx_model_name(model_name)
else:
    self.backend = "faster-qwen3-tts"
    # Non-MLX backends ignore mlx_quantization kwargs

On non-macOS systems, --qwen3_mlx_quantizations has no effect—the script falls back to CPU/GPU backends without quantization suffix support.

Memory and Cleanup

Each Qwen3TTSHandler instance calls cleanup() post-benchmark to:

• Remove temporary MLX audio cache files
• Release MLX GPU memory allocations
• Drop model weights from resident memory

This prevents cumulative memory pressure when iterating through multiple quantization levels.

Optional: Voice Cloning Benchmarks

Add --ref_audio /path/to/speaker.wav to evaluate quantization impact on voice cloning quality. The handler loads the reference through mlx_audio.tts.utils.load_model before synthesis, extending warm-up time proportionally.

---

Summary

• benchmark_tts.py provides unified CLI access to TTS quantization benchmarking on macOS
• --qwen3_mlx_quantizations accepts any subset of bf16, 4bit, 6bit, 8bit
• Qwen3TTSHandler auto-detects Apple Silicon and routes to MLX with suffix-appended model names
• Results JSON captures warm-up time, inference latency, and RTF for direct quantization comparison
• MLX quantization delivers 2-4× RTF improvements over bf16 with acceptable quality trade-offs

---

Frequently Asked Questions

What hardware is required to run TTS quantization benchmarks on macOS?

Apple Silicon Macs (M1/M2/M3/M4 series) with sufficient unified memory for the target model. The MLX backend leverages the Neural Engine and GPU cores; Intel Macs fall back to non-quantized CPU inference without the --qwen3_mlx_quantizations functionality.

Can I benchmark a single quantization level instead of all four?

Yes—pass only the desired quantization to the flag: --qwen3_mlx_quantizations 4bit. The normalize_qwen3_mlx_quantizations function validates against VALID_QWEN3_MLX_QUANTIZATIONS and accepts any non-empty subset.

How do I compare quantization quality subjectively?

The benchmark script measures speed, not perceptual quality. For quality evaluation, generate sample audio from each quantization level using Qwen3TTSHandler directly, then conduct ABX listening tests or compute objective metrics like Mel-Cepstral Distortion on reference utterances.

Why does warm-up time vary significantly across quantizations?

Lower-bit quantizations (4bit, 6bit, 8bit) use weight compression that reduces both disk I/O and memory bandwidth during model loading. The _apply_mlx_quantization_suffix function selects pre-quantized checkpoints from mlx-community/ that are smaller than the bf16 base model, directly reducing warmup_time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}