# TurboQuant Performance Benefits: Memory Reduction and Speed Optimization in MLX-VLM

> Discover TurboQuant's performance benefits, reducing MLX-VLM memory usage by up to 76% and accelerating attention computation with custom Metal kernels.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: performance
- Published: 2026-04-05

---

**TurboQuant compresses KV-cache entries from 16-bit to 2–4 bits per dimension, reducing memory usage by up to 76% and accelerating attention computation at long contexts through custom Metal kernels.**

TurboQuant is a high-performance KV-cache quantization scheme implemented in the `mlx-vlm` repository that enables vision-language models to handle longer contexts with significantly reduced memory overhead. By quantizing attention caches using specialized algorithms and custom GPU kernels, this quantization method delivers substantial memory savings while often improving inference speed for extended sequences.

## What Is TurboQuant?

TurboQuant is a **KV-cache quantization scheme** designed to reduce the memory footprint of attention mechanisms in transformer models. Unlike standard 16-bit floating-point storage, TurboQuant compresses cache entries to **2–4 bits per dimension** using random Hadamard rotation followed by MSE code-book quantization, as detailed in the repository's documentation.

## Memory Efficiency Benefits

### Aggressive Compression Ratios

According to the `mlx-vlm` source, TurboQuant reduces KV-cache memory consumption by **63% to 76%**. For example, when running Qwen3.5-4B, the KV-cache footprint drops from 4.1 GB to 0.97 GB, enabling much longer contexts on the same hardware configuration.

### Fractional-Bit Precision Support

TurboQuant supports **fractional-bit widths** such as 3.5 bits, which allocates 3 bits for keys and 4 bits for values. This granular control allows developers to trade minimal quality degradation for additional compression, tailoring memory usage to specific deployment constraints.

## Inference Speed Optimization

### Custom Metal Kernel Implementation

In [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), the implementation defines **custom Metal kernels** that operate directly on packed quantized data. These kernels fuse score computation with value aggregation, **avoiding full de-quantization** during the decoding phase, which significantly reduces memory-bandwidth bottlenecks.

### Long-Context Throughput Gains

Performance benchmarks in the repository demonstrate that per-layer attention becomes **faster than FP16 dense-attention** once the input context exceeds approximately **512k tokens**. At these lengths, the reduced bandwidth requirements outweigh the additional arithmetic overhead of quantized operations, resulting in higher tokens-per-second throughput.

## Seamless Integration Architecture

TurboQuant automatically quantizes global `KVCache` layers without requiring manual model modifications. The implementation in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) handles state definitions, quantization codecs, and kernel dispatch. Models utilizing memory-efficient cache variants such as rotating or MLA caches remain untouched, ensuring compatibility across diverse architectures.

## Implementation Across MLX-VLM

### Core Quantization Engine

The file [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) contains the core implementation, including quantization codecs, state management objects, and the Metal kernel interfaces that enable high-speed operations on compressed KV data.

### CLI and API Integration

The generation pipeline in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) and the server entry point in [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) parse the `--kv-bits` and `--kv-quant-scheme` arguments, forwarding these parameters to initialize TurboQuant-enabled caches automatically.

## Enabling TurboQuant in Practice

### Command-Line Usage

Enable TurboQuant with 3.5-bit quantization using the CLI:

```bash
mlx_vlm generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --prompt "Explain the implications of climate change on coastal cities."

```

### Python API Implementation

Programmatically configure TurboQuant when calling `generate`:

```python
from mlx_vlm import load, generate

model, processor = load("mlx-community/Qwen3.5-4B-4bit")
prompt = "Summarize the plot of The Matrix in one sentence."
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

```

### Server Deployment Mode

Deploy a FastAPI server with quantized KV-cache:

```bash
mlx_vlm server \
  --model google/gemma-4-31b-it \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant

```

## Summary

- TurboQuant reduces KV-cache memory usage by **63% to 76%** by compressing entries to 2–4 bits per dimension.
- **Custom Metal kernels** in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) avoid de-quantization overhead and fuse attention operations for reduced bandwidth usage.
- Inference speed exceeds FP16 performance at contexts longer than **512k tokens**.
- **Fractional-bit support** enables flexible precision tuning (e.g., 3.5 bits) for optimal quality-to-compression ratios.
- Seamless integration automatically quantizes standard `KVCache` layers while preserving specialized cache implementations.

## Frequently Asked Questions

### How much memory can TurboQuant save?

TurboQuant typically reduces KV-cache memory consumption by **63% to 76%**, depending on the selected bit-width. For instance, running Qwen3.5-4B with TurboQuant decreases the cache from 4.1 GB to 0.97 GB, as documented in the performance benchmarks.

### Does TurboQuant reduce inference speed?

At moderate context lengths, TurboQuant maintains competitive performance, but at sequences exceeding **512k tokens**, it becomes **faster than FP16 dense-attention**. The custom Metal kernels reduce memory-bandwidth pressure, which becomes the dominant bottleneck during long-context generation.

### What quantization bit-widths does TurboQuant support?

TurboQuant supports **2 to 4 bits per dimension**, including fractional configurations such as **3.5 bits** (3 bits for keys, 4 bits for values). This flexibility allows developers to balance compression ratios with model output quality.

### Is TurboQuant compatible with all models in MLX-VLM?

TurboQuant automatically integrates with models using standard global `KVCache` layers. However, it leaves untouched memory-efficient cache variants such as rotating caches or MLA (Multi-Head Latent Attention) caches to avoid interference with existing optimizations.