TurboQuant Performance Benefits: Memory Reduction and Speed Optimization in MLX-VLM

TurboQuant compresses KV-cache entries from 16-bit to 2–4 bits per dimension, reducing memory usage by up to 76% and accelerating attention computation at long contexts through custom Metal kernels.

TurboQuant is a high-performance KV-cache quantization scheme implemented in the mlx-vlm repository that enables vision-language models to handle longer contexts with significantly reduced memory overhead. By quantizing attention caches using specialized algorithms and custom GPU kernels, this quantization method delivers substantial memory savings while often improving inference speed for extended sequences.

What Is TurboQuant?

TurboQuant is a KV-cache quantization scheme designed to reduce the memory footprint of attention mechanisms in transformer models. Unlike standard 16-bit floating-point storage, TurboQuant compresses cache entries to 2–4 bits per dimension using random Hadamard rotation followed by MSE code-book quantization, as detailed in the repository's documentation.

Memory Efficiency Benefits

Aggressive Compression Ratios

According to the mlx-vlm source, TurboQuant reduces KV-cache memory consumption by 63% to 76%. For example, when running Qwen3.5-4B, the KV-cache footprint drops from 4.1 GB to 0.97 GB, enabling much longer contexts on the same hardware configuration.

Fractional-Bit Precision Support

TurboQuant supports fractional-bit widths such as 3.5 bits, which allocates 3 bits for keys and 4 bits for values. This granular control allows developers to trade minimal quality degradation for additional compression, tailoring memory usage to specific deployment constraints.

Inference Speed Optimization

Custom Metal Kernel Implementation

In mlx_vlm/turboquant.py, the implementation defines custom Metal kernels that operate directly on packed quantized data. These kernels fuse score computation with value aggregation, avoiding full de-quantization during the decoding phase, which significantly reduces memory-bandwidth bottlenecks.

Long-Context Throughput Gains

Performance benchmarks in the repository demonstrate that per-layer attention becomes faster than FP16 dense-attention once the input context exceeds approximately 512k tokens. At these lengths, the reduced bandwidth requirements outweigh the additional arithmetic overhead of quantized operations, resulting in higher tokens-per-second throughput.

Seamless Integration Architecture

TurboQuant automatically quantizes global KVCache layers without requiring manual model modifications. The implementation in mlx_vlm/turboquant.py handles state definitions, quantization codecs, and kernel dispatch. Models utilizing memory-efficient cache variants such as rotating or MLA caches remain untouched, ensuring compatibility across diverse architectures.

Implementation Across MLX-VLM

Core Quantization Engine

The file mlx_vlm/turboquant.py contains the core implementation, including quantization codecs, state management objects, and the Metal kernel interfaces that enable high-speed operations on compressed KV data.

CLI and API Integration

The generation pipeline in mlx_vlm/generate.py and the server entry point in mlx_vlm/server.py parse the --kv-bits and --kv-quant-scheme arguments, forwarding these parameters to initialize TurboQuant-enabled caches automatically.

Enabling TurboQuant in Practice

Command-Line Usage

Enable TurboQuant with 3.5-bit quantization using the CLI:

mlx_vlm generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --prompt "Explain the implications of climate change on coastal cities."

Python API Implementation

Programmatically configure TurboQuant when calling generate:

from mlx_vlm import load, generate

model, processor = load("mlx-community/Qwen3.5-4B-4bit")
prompt = "Summarize the plot of The Matrix in one sentence."
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

Server Deployment Mode

Deploy a FastAPI server with quantized KV-cache:

mlx_vlm server \
  --model google/gemma-4-31b-it \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant

Summary

• TurboQuant reduces KV-cache memory usage by 63% to 76% by compressing entries to 2–4 bits per dimension.
• Custom Metal kernels in mlx_vlm/turboquant.py avoid de-quantization overhead and fuse attention operations for reduced bandwidth usage.
• Inference speed exceeds FP16 performance at contexts longer than 512k tokens.
• Fractional-bit support enables flexible precision tuning (e.g., 3.5 bits) for optimal quality-to-compression ratios.
• Seamless integration automatically quantizes standard KVCache layers while preserving specialized cache implementations.

Frequently Asked Questions

How much memory can TurboQuant save?

TurboQuant typically reduces KV-cache memory consumption by 63% to 76%, depending on the selected bit-width. For instance, running Qwen3.5-4B with TurboQuant decreases the cache from 4.1 GB to 0.97 GB, as documented in the performance benchmarks.

Does TurboQuant reduce inference speed?

At moderate context lengths, TurboQuant maintains competitive performance, but at sequences exceeding 512k tokens, it becomes faster than FP16 dense-attention. The custom Metal kernels reduce memory-bandwidth pressure, which becomes the dominant bottleneck during long-context generation.

What quantization bit-widths does TurboQuant support?

TurboQuant supports 2 to 4 bits per dimension, including fractional configurations such as 3.5 bits (3 bits for keys, 4 bits for values). This flexibility allows developers to balance compression ratios with model output quality.

Is TurboQuant compatible with all models in MLX-VLM?

TurboQuant automatically integrates with models using standard global KVCache layers. However, it leaves untouched memory-efficient cache variants such as rotating caches or MLA (Multi-Head Latent Attention) caches to avoid interference with existing optimizations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}