# How to Benchmark TTS Quantization Variants on macOS: Complete Guide

> Benchmark TTS quantization variants on macOS with Qwen-3-TTS. Compare bf16, 4bit, 6bit, and 8bit levels by latency, RTF, and warm-up time. Run the benchmark script now.

- Repository: [Hugging Face/speech-to-speech](https://github.com/huggingface/speech-to-speech)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Run `python scripts/benchmark_tts.py --qwen3_mlx_quantizations bf16 4bit 6bit 8bit` on macOS to compare Qwen-3-TTS quantization levels by latency, RTF, and warm-up time.**

The **huggingface/speech-to-speech** repository ships with purpose-built tooling to evaluate Text-to-Speech (TTS) performance across quantization configurations. On Apple Silicon Macs, you can benchmark the MLX-optimized Qwen-3-TTS model at **bf16**, **4bit**, **6bit**, and **8bit** precisions using a single command-line invocation. This guide walks through the architecture, exact commands, and result interpretation.

---

## Architecture of the Benchmarking System

Three core components coordinate TTS quantization benchmarking on macOS:

| Component | Role | Source Location |
|-----------|------|---------------|
| [`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py) | CLI entry point, target builder, and metrics aggregator | [[`scripts/benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py)](https://github.com/huggingface/speech-to-speech/blob/main/scripts/benchmark_tts.py) |
| `Qwen3TTSHandler` | Platform detection, MLX model resolution, and quantization suffix application | [[`src/speech_to_speech/TTS/qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/TTS/qwen3_tts_handler.py) |
| [`mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/mlx_lock.py) | Serializes access to the MLX runtime across concurrent handler instances | [[`src/speech_to_speech/utils/mlx_lock.py`](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py)](https://github.com/huggingface/speech-to-speech/blob/main/src/speech_to_speech/utils/mlx_lock.py) |

### Key Functions and Constants

- **`VALID_QWEN3_MLX_QUANTIZATIONS`** — Hardcoded tuple at line 31 of [`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py) defining permitted values: `("4bit", "6bit", "8bit", "bf16")`
- **`normalize_qwen3_mlx_quantizations`** — Validates user input against the allowed set (lines 12-30)
- **`build_benchmark_targets`** — Expands `qwen3` into enumerated targets like `qwen3[4bit]` with per-quantization `handler_kwargs` (lines 34-51)
- **`_apply_mlx_quantization_suffix`** — Inserts or replaces quantization suffixes in MLX model identifiers (lines 42-52 of [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py))
- **`_resolve_mlx_model_name`** — Applies default 6-bit quantization when unspecified (lines 64-78)

---

## Running the Benchmark on macOS

### Prerequisites

```bash

# Clone repository

git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech

# Install MLX dependencies

pip install mlx-audio mlx-community

```

### Benchmark Command

```bash
python scripts/benchmark_tts.py \
  --text "Benchmarking Qwen-3-TTS quantization variants on Apple Silicon" \
  --handlers qwen3 \
  --iterations 10 \
  --qwen3_mlx_quantizations bf16 4bit 6bit 8bit \
  --output macos_quantization_bench.json

```

**Parameter breakdown:**

- `--handlers qwen3` — Selects the Qwen-3-TTS backend (auto-detects MLX on `darwin`)
- `--qwen3_mlx_quantizations` — Space-separated list from `VALID_QWEN3_MLX_QUANTIZATIONS`
- `--iterations 10` — Number of synthesis passes per quantization level (higher = more stable statistics)
- `--output` — Destination for structured JSON results

### Execution Flow

1. **CLI parsing** — `argparse` populates `args.qwen3_mlx_quantizations` with your quantization list
2. **Target expansion** — `build_benchmark_targets` generates four distinct benchmark targets:
   - `qwen3[bf16]` → `handler_kwargs: {"mlx_quantization": "bf16"}`
   - `qwen3[4bit]` → `handler_kwargs: {"mlx_quantization": "4bit"}`
   - `qwen3[6bit]` → `handler_kwargs: {"mlx_quantization": "6bit"}`
   - `qwen3[8bit]` → `handler_kwargs: {"mlx_quantization": "8bit"}`
3. **Handler instantiation** — For each target, `benchmark_handler` creates a `Qwen3TTSHandler`; `setup()` detects `platform == "darwin"` and routes to MLX
4. **Model resolution** — `_resolve_mlx_model_name` calls `_apply_mlx_quantization_suffix` to transform base model names (e.g., `mlx-community/Qwen3-TTS` → `mlx-community/Qwen3-TTS-4bit`)
5. **Streaming benchmark** — The loop at lines 165-200 measures:
   - **Warm-up time**: Model loading and weight decompression
   - **Time-to-first-chunk**: Latency before audio generation begins
   - **Total inference time**: End-to-end synthesis duration
   - **Real-time factor (RTF)**: Audio duration divided by inference time

---

## Understanding the Results

The JSON output follows this schema:

```json
{
  "timestamp": "2026-08-07 14:23:11",
  "results": [
    {
      "handler": "qwen3[bf16]",
      "warmup_time": 1.247,
      "avg_inference_time": 2.184,
      "min_inference_time": 2.091,
      "max_inference_time": 2.312,
      "std_inference_time": 0.076,
      "avg_audio_duration": 4.52,
      "avg_rtf": 2.07,
      "total_iterations": 10,
      "errors": []
    },
    {
      "handler": "qwen3[4bit]",
      "warmup_time": 0.823,
      "avg_inference_time": 1.156,
      "avg_rtf": 3.91,
      ...
    }
  ]
}

```

### Key Metrics for Quantization Comparison

| Metric | Interpretation | Optimization Target |
|--------|---------------|---------------------|
| `warmup_time` | Model loading overhead | Lower = faster cold-start |
| `avg_inference_time` | Mean synthesis latency | Lower = more responsive |
| `avg_rtf` | Throughput multiplier | Higher = better efficiency; **RTF > 1** means faster-than-real-time |

### Quantization Trade-offs

- **bf16**: Highest precision, largest memory footprint, slowest inference
- **8bit**: Modest compression with minimal quality degradation
- **6bit**: Balanced default (used when no quantization specified)
- **4bit**: Maximum speed and smallest size, potential quality reduction

Compare `avg_rtf` across entries: `qwen3[4bit]` typically achieves **~1.5-2× higher RTF** than `qwen3[bf16]`, representing substantial efficiency gains for production deployments.

---

## macOS-Specific Considerations

### Platform Enforcement

The quantization benchmarking path requires **Apple Silicon with MLX**. The detection logic in `Qwen3TTSHandler.setup()` (around line 90 of [`qwen3_tts_handler.py`](https://github.com/huggingface/speech-to-speech/blob/main/qwen3_tts_handler.py)) branches:

```python
if platform.system() == "darwin":
    self.backend = "mlx"
    self.model_name = self._resolve_mlx_model_name(model_name)
else:
    self.backend = "faster-qwen3-tts"
    # Non-MLX backends ignore mlx_quantization kwargs

```

On non-macOS systems, `--qwen3_mlx_quantizations` has no effect—the script falls back to CPU/GPU backends without quantization suffix support.

### Memory and Cleanup

Each `Qwen3TTSHandler` instance calls `cleanup()` post-benchmark to:

- Remove temporary MLX audio cache files
- Release MLX GPU memory allocations
- Drop model weights from resident memory

This prevents cumulative memory pressure when iterating through multiple quantization levels.

### Optional: Voice Cloning Benchmarks

Add `--ref_audio /path/to/speaker.wav` to evaluate quantization impact on voice cloning quality. The handler loads the reference through `mlx_audio.tts.utils.load_model` before synthesis, extending warm-up time proportionally.

---

## Summary

- **[`benchmark_tts.py`](https://github.com/huggingface/speech-to-speech/blob/main/benchmark_tts.py)** provides unified CLI access to TTS quantization benchmarking on macOS
- **`--qwen3_mlx_quantizations`** accepts any subset of `bf16`, `4bit`, `6bit`, `8bit`
- **`Qwen3TTSHandler`** auto-detects Apple Silicon and routes to MLX with suffix-appended model names
- Results JSON captures **warm-up time**, **inference latency**, and **RTF** for direct quantization comparison
- MLX quantization delivers **2-4× RTF improvements** over bf16 with acceptable quality trade-offs

---

## Frequently Asked Questions

### What hardware is required to run TTS quantization benchmarks on macOS?

Apple Silicon Macs (M1/M2/M3/M4 series) with sufficient unified memory for the target model. The MLX backend leverages the Neural Engine and GPU cores; Intel Macs fall back to non-quantized CPU inference without the `--qwen3_mlx_quantizations` functionality.

### Can I benchmark a single quantization level instead of all four?

Yes—pass only the desired quantization to the flag: `--qwen3_mlx_quantizations 4bit`. The `normalize_qwen3_mlx_quantizations` function validates against `VALID_QWEN3_MLX_QUANTIZATIONS` and accepts any non-empty subset.

### How do I compare quantization quality subjectively?

The benchmark script measures speed, not perceptual quality. For quality evaluation, generate sample audio from each quantization level using `Qwen3TTSHandler` directly, then conduct ABX listening tests or compute objective metrics like Mel-Cepstral Distortion on reference utterances.

### Why does warm-up time vary significantly across quantizations?

Lower-bit quantizations (4bit, 6bit, 8bit) use weight compression that reduces both disk I/O and memory bandwidth during model loading. The `_apply_mlx_quantization_suffix` function selects pre-quantized checkpoints from `mlx-community/` that are smaller than the bf16 base model, directly reducing `warmup_time`.