# TurboQuant KV Cache Quantization in MLX-VLM: Low-Bit Compression for Efficient Generation

> Discover TurboQuant KV Cache Quantization in MLX-VLM. This low-bit compression scheme efficiently reduces memory usage during generation, maintaining attention score fidelity.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: deep-dive
- Published: 2026-04-05

---

**TurboQuant KV Cache Quantization is a low-bit integer compression scheme that quantizes transformer key-value tensors on-the-fly to configurable bit widths, reducing memory usage during autoregressive generation while maintaining attention score fidelity.**

MLX-VLM, the Apple MLX-based vision-language model framework, implements **TurboQuant KV Cache Quantization** to address the memory bottleneck of growing KV caches during long-context inference. Unlike standard full-precision caches that store `float16` tensors, TurboQuant compresses keys and values to fractional bit widths (e.g., 3.5 bits) using statistical quantization and GPU-accelerated codecs.

## Architecture of TurboQuant KV Cache Quantization

The implementation centers on the `TurboQuantKVCache` class in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), which extends the base cache interface to provide quantized storage and retrieval.

### The TurboQuantKVCache Class

At lines 4790–4812 of [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), the `TurboQuantKVCache` class inherits from `_BaseCache` and manages quantized KV states. It stores tensors in packed `uint32` buffers alongside per-token norms, retaining the original tensor shape for efficient slicing while compressing the underlying data.

```python
from mlx_vlm.turboquant import TurboQuantKVCache

# Initialize cache with 3.5-bit quantization

cache = TurboQuantKVCache(bits=3.5)

```

### Quantization Codecs and Bit Splitting

The `_ensure_codecs` method (around line 19 in [`turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/turboquant.py)) constructs MSE-optimized integer codecs for keys and values. For fractional bit widths, the implementation splits bits between key and value tensors—assigning `floor(bits)` to keys and `ceil(bits)` to values—to achieve the target compression rate.

### Fused vs. Separate Quantization Paths

TurboQuant optimizes throughput through `_try_fused_kv_quantize` (line 40), a Metal kernel that packs key and value tensors together when both have singleton head dimensions, halving dispatch overhead. When shapes prevent fusion, the system falls back to separate quantization via `self.key_codec.quantize` and `self.value_codec.quantize` within the `update_and_fetch` method.

## How TurboQuant KV Cache Quantization Works

TurboQuant achieves high fidelity at low bit widths through three statistical techniques:

**Random Hadamard Transform (Rotation)** – Before quantization, vectors are multiplied by a deterministic rotation matrix (`_rotation_matrix`) to distribute information uniformly across dimensions, making low-bit scalar quantization effective.

**Beta-PDF Lloyd-Max Quantization** – The `_TurboQuantMSECodec` builds a codebook matching the KV vector distribution using a Beta-probability-density-function-based Lloyd-max quantizer. This preserves the most informative directions while discarding redundant precision.

**MSE-Optimized Decoding** – The codec minimizes mean-square error for the given bit budget. During attention scoring, Metal kernels such as `_metal_mse_score` (line 58) compute scores directly from the compressed representation without fully expanding tensors to `float32`.

## Using TurboQuant KV Cache Quantization in MLX-VLM

Integration occurs primarily through the generation pipeline in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py), with automatic codec selection based on CLI arguments.

### Enabling TurboQuant via CLI

Users trigger quantization using the `--kv-bits` and `--kv-quant-scheme` flags. Non-integer bit widths automatically select the TurboQuant scheme:

```bash
python -m mlx_vlm.main \
  --model mlx-community/nanoLLaVA-1.5-8bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --max-tokens 128

```

Argument parsing occurs at lines 29–38 of [`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py), while cache instantiation happens at lines 255–268 via `maybe_quantize_kv_cache`.

### Programmatic API Integration

For custom generation loops, instantiate `TurboQuantKVCache` and pass it through `generate_step`:

```python
from mlx_vlm.generate import generate_step
from mlx_vlm.turboquant import TurboQuantKVCache

def generate_with_quantization(model, input_ids, kv_bits=4.0):
    cache = TurboQuantKVCache(bits=kv_bits)
    
    generator = generate_step(
        input_ids=input_ids,
        model=model,
        kv_bits=kv_bits,
        kv_quant_scheme="turboquant",
        kv_group_size=64,
        quantized_kv_start=5000,
        max_tokens=20,
    )
    
    for token, _ in generator:
        yield token

```

### Manual Cache Operations

For debugging or custom attention implementations, manually update and inspect the cache:

```python

# After forward pass, quantize new KV tensors

kv_state_keys, kv_state_values = cache.update_and_fetch(raw_keys, raw_values)

# Inspect packed representation (uint32 buffers)

print("Packed shape:", cache.keys.shape)

# Dequantize for inspection

keys_fp, values_fp = cache.dequantize(kv_state_keys, kv_state_values)

```

The packed tensors use `uint32` words where each word encodes multiple low-bit values according to the configured bit width.

## Summary

- **TurboQuantKVCache** in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) provides the core quantized cache implementation, storing KV tensors in packed `uint32` buffers with per-token norms.
- **Bit splitting** allows fractional quantization (e.g., 3.5 bits) by distributing bits unevenly between keys and values using floor/ceil allocation in `_ensure_codecs`.
- **Fused Metal kernels** optimize performance when head dimensions permit combined key-value quantization via `_try_fused_kv_quantize`, falling back to separate paths for incompatible shapes.
- **Statistical techniques** including Random Hadamard Transform and Beta-PDF Lloyd-Max quantization preserve attention fidelity at low bit widths through the `_TurboQuantMSECodec`.
- **CLI flags** `--kv-bits` and `--kv-quant-scheme turboquant` enable the feature in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py), with automatic detection of fractional bit widths triggering TurboQuant selection.

## Frequently Asked Questions

### What bit widths does TurboQuant KV Cache Quantization support?

TurboQuant supports arbitrary fractional bit widths (e.g., 3.5, 4.2) in addition to integer values. The `_ensure_codecs` method automatically splits fractional bits between keys and values—typically assigning the floor to keys and ceiling to values—to achieve the target compression rate while maintaining balanced quality.

### How does TurboQuant affect generation quality compared to full-precision caches?

According to the MSE-optimized codec implementation in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), TurboQuant minimizes mean-square error through statistical quantization and random rotation. The Metal kernels compute attention scores directly from compressed representations, preserving fidelity while reducing memory usage by 4–5x at 3.5-bit precision compared to `float16`.

### Can I use TurboQuant with any vision-language model in MLX-VLM?

TurboQuant works with any transformer model using the standard MLX-VLM cache interface. The `maybe_quantize_kv_cache` wrapper in [`generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/generate.py) (imported at line 14) automatically swaps in `TurboQuantKVCache` when `--kv-bits` is specified and the scheme is set to `turboquant`, requiring no model architecture changes.

### Where is the dequantization logic implemented?

The `dequantize` method of `TurboQuantKVCache` (in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py)) handles reconstruction of full-precision tensors. It uses the stored rotation matrix and codebooks to decode packed `uint32` buffers back to `float32` on-the-fly during attention scoring or when explicit retrieval is requested.