How to Use TurboQuant with the MLX-VLM CLI: KV-Cache Compression Guide

Use the --kv-bits flag with a fractional value (e.g., 3.5) or pass --kv-quant-scheme turboquant to enable TurboQuant KV-cache quantization in the MLX-VLM CLI, reducing memory usage by up to 76% while preserving generation quality.

TurboQuant is a deterministic KV-cache quantization backend available in the Blaizzy/mlx-vlm repository that compresses attention memory to 2-4 bits per dimension. When you use TurboQuant with the MLX-VLM CLI, the system applies random rotation and codebook quantization to dramatically reduce memory footprint during inference on Apple Silicon devices. This guide covers the CLI flags, activation logic, and implementation details needed to deploy TurboQuant effectively.

What Is TurboQuant?

TurboQuant is a KV-cache quantization backend that reduces attention memory usage by applying deterministic random rotation to KV vectors followed by codebook quantization. According to the source code in [mlx_vlm/turboquant.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), the implementation uses rotation helpers (_rotation_matrix, _rht_forward, _rht_inverse) and a codebook generator (_codebook) to pack low-bit data without full de-quantization passes.

The quantization kernels (_mse_score_kernel, _prod_score_repeat_kernel) are generated on-the-fly with Metal and fuse scoring directly on packed data. This design achieves approximately 76% memory reduction when using 3.5-bit quantization compared to full-precision caches.

Activation Logic and CLI Integration

The MLX-VLM CLI determines whether to use TurboQuant through the turboquant_enabled function in mlx_vlm/turboquant.py. This helper checks two conditions:

1. Fractional bits auto-select: If --kv-bits contains a fractional value (e.g., 3.5), TurboQuant activates automatically.
2. Explicit scheme flag: If --kv-quant-scheme turboquant is passed, TurboQuant activates regardless of bit width.

# mlx_vlm/turboquant.py

def turboquant_enabled(bits: Optional[float], scheme: Optional[str] = None) -> bool:
    if bits is None:
        return False
    if scheme == "turboquant":
        return True
    bits = float(bits)
    return not math.isclose(bits, round(bits), abs_tol=1e-6)

Both flags are registered in the argument parser in [mlx_vlm/generate.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py#L29-L41) and propagated to the server entry point in [mlx_vlm/server.py](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py#L1415-L1419).

Cache Quantization Hook

During generation, the maybe_quantize_kv_cache function in mlx_vlm/generate.py examines each layer's cache. If TurboQuant is enabled, it replaces standard KVCache instances with TurboQuantKVCache objects:

# mlx_vlm/generate.py

def maybe_quantize_kv_cache(...):
    ...
    if turboquant_enabled(kv_bits, kv_quant_scheme):
        def quantize_entry(entry):
            if isinstance(entry, TurboQuantKVCache):
                return entry
            if isinstance(entry, cache.RotatingKVCache):
                return entry
            if isinstance(entry, cache.KVCache):
                if entry.offset == 0:
                    return TurboQuantKVCache(bits=kv_bits)
                if entry.offset < quantized_kv_start:
                    return entry
                return TurboQuantKVCache.from_cache(entry, bits=kv_bits)
            ...
        # Skip the last layer (highly sensitive to quantization)

        for index, layer_cache in enumerate(prompt_cache):
            if index == last_idx:
                continue
            prompt_cache[index] = quantize_entry(layer_cache)
        return
    # Fallback to uniform quantizer

    mlx_maybe_quantize_kv_cache(...)

Note that the implementation deliberately skips the last layer because it is highly sensitive to quantization artifacts.

CLI Usage Examples

Automatic Selection with Fractional Bits

To use TurboQuant with the MLX-VLM CLI using automatic detection, provide a fractional value for --kv-bits:

mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --prompt "Explain the theory of relativity in simple terms."

The fractional value 3.5 triggers the turboquant_enabled check and activates TurboQuant automatically.

Explicit Activation with Integer Bits

To force TurboQuant even with integer bit widths, use the --kv-quant-scheme flag:

mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 4 \
  --kv-quant-scheme turboquant \
  --prompt "Summarize the plot of Inception."

This approach is useful when you need predictable bit-width memory layouts but still want the rotation-based quantization benefits.

Server Deployment

Deploy the FastAPI server with TurboQuant compression enabled for all incoming requests:

mlx_vlm.server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --port 8080

All requests to http://localhost:8080/v1/chat/completions will utilize the compressed KV cache.

Python API Usage

You can also access TurboQuant programmatically using the same parameters:

from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen3.5-4B-4bit",
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
)

prompt = "What are the health benefits of a Mediterranean diet?"
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

Summary

• Use --kv-bits with fractional values (e.g., 3.5) to automatically select TurboQuant when running the MLX-VLM CLI.
• Force explicit activation by passing --kv-quant-scheme turboquant alongside any valid bit width.
• Memory efficiency: TurboQuant achieves approximately 76% memory reduction compared to full-precision caches by using deterministic rotation and codebook quantization.
• Layer protection: The implementation in mlx_vlm/generate.py preserves the last transformer layer in full precision to maintain generation quality.
• Universal support: These flags work in both the CLI generator (mlx_vlm.generate) and the server deployment (mlx_vlm.server).

Frequently Asked Questions

What is the difference between TurboQuant and uniform quantization?

TurboQuant applies deterministic random rotation to KV vectors followed by codebook quantization, while uniform quantization uses standard linear quantization buckets. TurboQuant is automatically selected when you specify fractional bit widths (e.g., 3.5) or explicitly request it via --kv-quant-scheme turboquant, whereas uniform quantization is the default fallback for integer bit widths.

Why does TurboQuant skip the last layer?

The maybe_quantize_kv_cache function in mlx_vlm/generate.py intentionally skips quantizing the final transformer layer because it is highly sensitive to quantization errors. Preserving full precision in this layer helps maintain overall generation quality while still achieving significant memory savings from compressing the earlier layers.

Can I use TurboQuant with the MLX-VLM server?

Yes. The server entry point in mlx_vlm/server.py (lines 1415-1419) propagates both --kv-bits and --kv-quant-scheme arguments to the underlying generation engine. Start the server with these flags to apply TurboQuant compression to all chat completion requests.

What bit widths does TurboQuant support?

TurboQuant supports 2-4 bits per dimension for KV-cache compression. You can specify fractional values like 3.5 bits to fine-tune the memory-quality tradeoff. The system uses the turboquant_enabled function to validate that the requested configuration activates the TurboQuant backend rather than uniform quantization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}