# How to Use TurboQuant with the MLX-VLM Python API: A Complete Guide

> Learn how to use TurboQuant with the MLX-VLM Python API to compress KV-caches to 2-4 bits per dimension. This guide shows you how to leverage fractional kv_bits or kv_quant_scheme turboquant for efficient model performance.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**TurboQuant compresses the KV-cache to 2-4 bits per dimension using rotation-based codebook quantization, activated by passing fractional `kv_bits` values or setting `kv_quant_scheme="turboquant"` in the MLX-VLM Python API.**

The MLX-VLM library supports advanced KV-cache compression through TurboQuant, a backend that reduces memory usage by up to 76% while preserving generation quality. This guide explains how to enable and configure TurboQuant using the Python API, referencing the actual implementation in the `Blaizzy/mlx-vlm` repository.

## What is TurboQuant?

TurboQuant is a **KV-cache quantization backend** that compresses attention memory to 2-4 bits per dimension while maintaining generation quality. It applies a deterministic random rotation to KV vectors followed by **codebook quantization**, implemented in **[[`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py)**.

The core implementation includes:

- **`_rotation_matrix`**, **`_rht_forward`**, and **`_rht_inverse`** – Helpers for the random Hadamard transform that rotates KV vectors before quantization.
- **`_codebook`** – Generates the quantization codebook used to compress the rotated values.
- **Metal kernel fusion** – Functions like `_mse_score_kernel` and `_prod_score_repeat_kernel` are generated on-the-fly to fuse scoring directly on packed low-bit data, avoiding full dequantization passes.

## How TurboQuant Activation Works

The library automatically selects TurboQuant when you supply a **fractional** value for `kv_bits` (e.g., `3.5`). You can also force it explicitly using the `kv_quant_scheme` parameter. The selection logic lives in the `turboquant_enabled` helper:

```python

# mlx_vlm/turboquant.py

def turboquant_enabled(bits: Optional[float], scheme: Optional[str] = None) -> bool:
    if bits is None:
        return False
    if scheme == "turboquant":
        return True
    bits = float(bits)
    return not math.isclose(bits, round(bits), abs_tol=1e-6)

```

This function returns `True` when the bit width is not an integer (within tolerance) or when the scheme is explicitly set to `"turboquant"`.

## Using TurboQuant in the MLX-VLM Python API

To activate TurboQuant programmatically, pass the `kv_bits` and optional `kv_quant_scheme` parameters to both the `load()` and `generate()` functions:

```python
from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen3.5-4B-4bit",
    kv_bits=3.5,                     # Fractional value activates TurboQuant

    kv_quant_scheme="turboquant",    # Explicit enforcement (optional)

)

prompt = "Explain quantum computing in simple terms."
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

```

**Key parameters:**

- **`kv_bits`** – Target bit width per dimension. Values like `3.5`, `2.5`, or `4.0` (with fractional scheme) trigger TurboQuant.
- **`kv_quant_scheme`** – Set to `"turboquant"` to force the backend even with integer bit widths, or `"uniform"` for standard quantization.

## CLI and Server Configuration

TurboQuant is also available via command-line tools. The flags are defined in **[[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)** and propagated to the server in **[[`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py)**.

**Activate via fractional bits (auto-selection):**

```bash
mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --prompt "Summarize the theory of relativity."

```

**Explicit activation with integer bits:**

```bash
mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 4 \
  --kv-quant-scheme turboquant \
  --prompt "Describe the water cycle."

```

**FastAPI server with TurboQuant:**

```bash
mlx_vlm.server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --port 8080

```

Requests to `http://localhost:8080/v1/chat/completions` will use the compressed KV cache.

## Implementation Details: Cache Substitution

During generation, the **`maybe_quantize_kv_cache`** function in **[[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)** examines each layer's cache and substitutes `KVCache` instances with `TurboQuantKVCache` when TurboQuant is enabled:

```python

# mlx_vlm/generate.py

def maybe_quantize_kv_cache(...):
    ...
    if turboquant_enabled(kv_bits, kv_quant_scheme):
        def quantize_entry(entry):
            if isinstance(entry, TurboQuantKVCache):
                return entry
            if isinstance(entry, cache.RotatingKVCache):
                return entry
            if isinstance(entry, cache.KVCache):
                if entry.offset == 0:
                    return TurboQuantKVCache(bits=kv_bits)
                if entry.offset < quantized_kv_start:
                    return entry
                return TurboQuantKVCache.from_cache(entry, bits=kv_bits)
            ...
        # Skip the last layer (highly sensitive to quantization)

        for index, layer_cache in enumerate(prompt_cache):
            if index == last_idx:
                continue
            prompt_cache[index] = quantize_entry(layer_cache)
        return
    # Fallback to uniform quantizer

    mlx_maybe_quantize_kv_cache(...)

```

**Critical behaviors:**

- **Empty cache optimization** – If `entry.offset == 0`, it creates a fresh `TurboQuantKVCache` instead of converting.
- **Last layer protection** – The final layer is skipped because it is highly sensitive to quantization errors.
- **Rotating cache preservation** – Existing `RotatingKVCache` instances are left untouched.

## Summary

- **TurboQuant** provides 2-4 bit KV-cache compression via rotation and codebook quantization in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py).
- **Automatic activation** occurs when passing fractional `kv_bits` (e.g., `3.5`) to `load()` or `generate()`.
- **Explicit control** is available via `kv_quant_scheme="turboquant"` for integer bit widths or forced selection.
- **Memory savings** reach approximately 76% with 3.5-bit quantization, enabled by fused Metal kernels that avoid dequantization overhead.
- **Implementation** substitutes `TurboQuantKVCache` for standard `KVCache` in all layers except the final one during the forward pass.

## Frequently Asked Questions

### What bit widths does TurboQuant support?

TurboQuant supports **2 to 4 bits per dimension**, including fractional values like `2.5`, `3.5`, or `3.9`. Fractional bit widths automatically trigger the TurboQuant backend in the MLX-VLM Python API.

### Can I use TurboQuant with the FastAPI server?

Yes. Start the server using `mlx_vlm.server` with the `--kv-bits` and `--kv-quant-scheme turboquant` flags. All subsequent requests to the `/v1/chat/completions` endpoint will utilize the compressed KV cache.

### Why is the last layer skipped during TurboQuant quantization?

The implementation explicitly excludes the final layer from quantization because it is **highly sensitive to compression artifacts**. Skipping this layer preserves generation quality while still achieving significant memory savings from quantizing earlier layers.

### How much memory does TurboQuant actually save?

According to the MLX-VLM implementation, TurboQuant reduces KV-cache memory usage by approximately **76% when using 3.5-bit compression**, allowing much longer context windows on Apple Silicon devices.