How to Use TurboQuant with the MLX-VLM Python API: A Complete Guide

TurboQuant compresses the KV-cache to 2-4 bits per dimension using rotation-based codebook quantization, activated by passing fractional kv_bits values or setting kv_quant_scheme="turboquant" in the MLX-VLM Python API.

The MLX-VLM library supports advanced KV-cache compression through TurboQuant, a backend that reduces memory usage by up to 76% while preserving generation quality. This guide explains how to enable and configure TurboQuant using the Python API, referencing the actual implementation in the Blaizzy/mlx-vlm repository.

What is TurboQuant?

TurboQuant is a KV-cache quantization backend that compresses attention memory to 2-4 bits per dimension while maintaining generation quality. It applies a deterministic random rotation to KV vectors followed by codebook quantization, implemented in [mlx_vlm/turboquant.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py).

The core implementation includes:

• _rotation_matrix, _rht_forward, and _rht_inverse – Helpers for the random Hadamard transform that rotates KV vectors before quantization.
• _codebook – Generates the quantization codebook used to compress the rotated values.
• Metal kernel fusion – Functions like _mse_score_kernel and _prod_score_repeat_kernel are generated on-the-fly to fuse scoring directly on packed low-bit data, avoiding full dequantization passes.

How TurboQuant Activation Works

The library automatically selects TurboQuant when you supply a fractional value for kv_bits (e.g., 3.5). You can also force it explicitly using the kv_quant_scheme parameter. The selection logic lives in the turboquant_enabled helper:

# mlx_vlm/turboquant.py

def turboquant_enabled(bits: Optional[float], scheme: Optional[str] = None) -> bool:
    if bits is None:
        return False
    if scheme == "turboquant":
        return True
    bits = float(bits)
    return not math.isclose(bits, round(bits), abs_tol=1e-6)

This function returns True when the bit width is not an integer (within tolerance) or when the scheme is explicitly set to "turboquant".

Using TurboQuant in the MLX-VLM Python API

To activate TurboQuant programmatically, pass the kv_bits and optional kv_quant_scheme parameters to both the load() and generate() functions:

from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen3.5-4B-4bit",
    kv_bits=3.5,                     # Fractional value activates TurboQuant

    kv_quant_scheme="turboquant",    # Explicit enforcement (optional)

)

prompt = "Explain quantum computing in simple terms."
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

Key parameters:

• kv_bits – Target bit width per dimension. Values like 3.5, 2.5, or 4.0 (with fractional scheme) trigger TurboQuant.
• kv_quant_scheme – Set to "turboquant" to force the backend even with integer bit widths, or "uniform" for standard quantization.

CLI and Server Configuration

TurboQuant is also available via command-line tools. The flags are defined in [mlx_vlm/generate.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) and propagated to the server in [mlx_vlm/server.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py).

Activate via fractional bits (auto-selection):

mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --prompt "Summarize the theory of relativity."

Explicit activation with integer bits:

mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 4 \
  --kv-quant-scheme turboquant \
  --prompt "Describe the water cycle."

FastAPI server with TurboQuant:

mlx_vlm.server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --port 8080

Requests to http://localhost:8080/v1/chat/completions will use the compressed KV cache.

Implementation Details: Cache Substitution

During generation, the maybe_quantize_kv_cache function in [mlx_vlm/generate.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) examines each layer's cache and substitutes KVCache instances with TurboQuantKVCache when TurboQuant is enabled:

# mlx_vlm/generate.py

def maybe_quantize_kv_cache(...):
    ...
    if turboquant_enabled(kv_bits, kv_quant_scheme):
        def quantize_entry(entry):
            if isinstance(entry, TurboQuantKVCache):
                return entry
            if isinstance(entry, cache.RotatingKVCache):
                return entry
            if isinstance(entry, cache.KVCache):
                if entry.offset == 0:
                    return TurboQuantKVCache(bits=kv_bits)
                if entry.offset < quantized_kv_start:
                    return entry
                return TurboQuantKVCache.from_cache(entry, bits=kv_bits)
            ...
        # Skip the last layer (highly sensitive to quantization)

        for index, layer_cache in enumerate(prompt_cache):
            if index == last_idx:
                continue
            prompt_cache[index] = quantize_entry(layer_cache)
        return
    # Fallback to uniform quantizer

    mlx_maybe_quantize_kv_cache(...)

Critical behaviors:

• Empty cache optimization – If entry.offset == 0, it creates a fresh TurboQuantKVCache instead of converting.
• Last layer protection – The final layer is skipped because it is highly sensitive to quantization errors.
• Rotating cache preservation – Existing RotatingKVCache instances are left untouched.

Summary

• TurboQuant provides 2-4 bit KV-cache compression via rotation and codebook quantization in mlx_vlm/turboquant.py.
• Automatic activation occurs when passing fractional kv_bits (e.g., 3.5) to load() or generate().
• Explicit control is available via kv_quant_scheme="turboquant" for integer bit widths or forced selection.
• Memory savings reach approximately 76% with 3.5-bit quantization, enabled by fused Metal kernels that avoid dequantization overhead.
• Implementation substitutes TurboQuantKVCache for standard KVCache in all layers except the final one during the forward pass.

Frequently Asked Questions

What bit widths does TurboQuant support?

TurboQuant supports 2 to 4 bits per dimension, including fractional values like 2.5, 3.5, or 3.9. Fractional bit widths automatically trigger the TurboQuant backend in the MLX-VLM Python API.

Can I use TurboQuant with the FastAPI server?

Yes. Start the server using mlx_vlm.server with the --kv-bits and --kv-quant-scheme turboquant flags. All subsequent requests to the /v1/chat/completions endpoint will utilize the compressed KV cache.

Why is the last layer skipped during TurboQuant quantization?

The implementation explicitly excludes the final layer from quantization because it is highly sensitive to compression artifacts. Skipping this layer preserves generation quality while still achieving significant memory savings from quantizing earlier layers.

How much memory does TurboQuant actually save?

According to the MLX-VLM implementation, TurboQuant reduces KV-cache memory usage by approximately 76% when using 3.5-bit compression, allowing much longer context windows on Apple Silicon devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}