# How to Use TurboQuant with the MLX-VLM CLI: KV-Cache Compression Guide

> Learn to use TurboQuant with the MLX-VLM CLI using --kv-bits or --kv-quant-scheme turboquant. Reduce memory by up to 76% without compromising quality.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**Use the `--kv-bits` flag with a fractional value (e.g., `3.5`) or pass `--kv-quant-scheme turboquant` to enable TurboQuant KV-cache quantization in the MLX-VLM CLI, reducing memory usage by up to 76% while preserving generation quality.**

TurboQuant is a deterministic KV-cache quantization backend available in the [Blaizzy/mlx-vlm](https://github.com/Blaizzy/mlx-vlm) repository that compresses attention memory to 2-4 bits per dimension. When you use TurboQuant with the MLX-VLM CLI, the system applies random rotation and codebook quantization to dramatically reduce memory footprint during inference on Apple Silicon devices. This guide covers the CLI flags, activation logic, and implementation details needed to deploy TurboQuant effectively.

## What Is TurboQuant?

TurboQuant is a **KV-cache quantization backend** that reduces attention memory usage by applying deterministic random rotation to KV vectors followed by **codebook quantization**. According to the source code in [[`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py), the implementation uses rotation helpers (`_rotation_matrix`, `_rht_forward`, `_rht_inverse`) and a codebook generator (`_codebook`) to pack low-bit data without full de-quantization passes.

The quantization kernels (`_mse_score_kernel`, `_prod_score_repeat_kernel`) are generated on-the-fly with Metal and fuse scoring directly on packed data. This design achieves approximately **76% memory reduction** when using 3.5-bit quantization compared to full-precision caches.

## Activation Logic and CLI Integration

The MLX-VLM CLI determines whether to use TurboQuant through the `turboquant_enabled` function in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py). This helper checks two conditions:

1. **Fractional bits auto-select**: If `--kv-bits` contains a fractional value (e.g., `3.5`), TurboQuant activates automatically.
2. **Explicit scheme flag**: If `--kv-quant-scheme turboquant` is passed, TurboQuant activates regardless of bit width.

```python

# mlx_vlm/turboquant.py

def turboquant_enabled(bits: Optional[float], scheme: Optional[str] = None) -> bool:
    if bits is None:
        return False
    if scheme == "turboquant":
        return True
    bits = float(bits)
    return not math.isclose(bits, round(bits), abs_tol=1e-6)

```

Both flags are registered in the argument parser in [[`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py#L29-L41) and propagated to the server entry point in [[`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py)](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py#L1415-L1419).

## Cache Quantization Hook

During generation, the `maybe_quantize_kv_cache` function in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) examines each layer's cache. If TurboQuant is enabled, it replaces standard `KVCache` instances with `TurboQuantKVCache` objects:

```python

# mlx_vlm/generate.py

def maybe_quantize_kv_cache(...):
    ...
    if turboquant_enabled(kv_bits, kv_quant_scheme):
        def quantize_entry(entry):
            if isinstance(entry, TurboQuantKVCache):
                return entry
            if isinstance(entry, cache.RotatingKVCache):
                return entry
            if isinstance(entry, cache.KVCache):
                if entry.offset == 0:
                    return TurboQuantKVCache(bits=kv_bits)
                if entry.offset < quantized_kv_start:
                    return entry
                return TurboQuantKVCache.from_cache(entry, bits=kv_bits)
            ...
        # Skip the last layer (highly sensitive to quantization)

        for index, layer_cache in enumerate(prompt_cache):
            if index == last_idx:
                continue
            prompt_cache[index] = quantize_entry(layer_cache)
        return
    # Fallback to uniform quantizer

    mlx_maybe_quantize_kv_cache(...)

```

Note that the implementation deliberately **skips the last layer** because it is highly sensitive to quantization artifacts.

## CLI Usage Examples

### Automatic Selection with Fractional Bits

To use TurboQuant with the MLX-VLM CLI using automatic detection, provide a fractional value for `--kv-bits`:

```bash
mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --prompt "Explain the theory of relativity in simple terms."

```

The fractional value `3.5` triggers the `turboquant_enabled` check and activates TurboQuant automatically.

### Explicit Activation with Integer Bits

To force TurboQuant even with integer bit widths, use the `--kv-quant-scheme` flag:

```bash
mlx_vlm.generate \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 4 \
  --kv-quant-scheme turboquant \
  --prompt "Summarize the plot of Inception."

```

This approach is useful when you need predictable bit-width memory layouts but still want the rotation-based quantization benefits.

### Server Deployment

Deploy the FastAPI server with TurboQuant compression enabled for all incoming requests:

```bash
mlx_vlm.server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --port 8080

```

All requests to `http://localhost:8080/v1/chat/completions` will utilize the compressed KV cache.

### Python API Usage

You can also access TurboQuant programmatically using the same parameters:

```python
from mlx_vlm import load, generate

model, processor = load(
    "mlx-community/Qwen3.5-4B-4bit",
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
)

prompt = "What are the health benefits of a Mediterranean diet?"
output = generate(
    model,
    processor,
    prompt,
    kv_bits=3.5,
    kv_quant_scheme="turboquant",
    max_tokens=200,
)
print(output)

```

## Summary

- **Use `--kv-bits` with fractional values** (e.g., `3.5`) to automatically select TurboQuant when running the MLX-VLM CLI.
- **Force explicit activation** by passing `--kv-quant-scheme turboquant` alongside any valid bit width.
- **Memory efficiency**: TurboQuant achieves approximately 76% memory reduction compared to full-precision caches by using deterministic rotation and codebook quantization.
- **Layer protection**: The implementation in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) preserves the last transformer layer in full precision to maintain generation quality.
- **Universal support**: These flags work in both the CLI generator (`mlx_vlm.generate`) and the server deployment (`mlx_vlm.server`).

## Frequently Asked Questions

### What is the difference between TurboQuant and uniform quantization?

TurboQuant applies **deterministic random rotation** to KV vectors followed by codebook quantization, while uniform quantization uses standard linear quantization buckets. TurboQuant is automatically selected when you specify fractional bit widths (e.g., `3.5`) or explicitly request it via `--kv-quant-scheme turboquant`, whereas uniform quantization is the default fallback for integer bit widths.

### Why does TurboQuant skip the last layer?

The `maybe_quantize_kv_cache` function in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) intentionally skips quantizing the final transformer layer because it is **highly sensitive to quantization errors**. Preserving full precision in this layer helps maintain overall generation quality while still achieving significant memory savings from compressing the earlier layers.

### Can I use TurboQuant with the MLX-VLM server?

Yes. The server entry point in [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) (lines 1415-1419) propagates both `--kv-bits` and `--kv-quant-scheme` arguments to the underlying generation engine. Start the server with these flags to apply TurboQuant compression to all chat completion requests.

### What bit widths does TurboQuant support?

TurboQuant supports **2-4 bits per dimension** for KV-cache compression. You can specify fractional values like `3.5` bits to fine-tune the memory-quality tradeoff. The system uses the `turboquant_enabled` function to validate that the requested configuration activates the TurboQuant backend rather than uniform quantization.