# How to Configure TurboQuant for the MLX-VLM Server: Complete Setup Guide

> Learn how to configure TurboQuant for the MLX-VLM server. Set non-integer bit-width and quantization scheme for optimal performance. Complete setup guide.

- Repository: [Prince Canuma/mlx-vlm](https://github.com/Blaizzy/mlx-vlm)
- Tags: how-to-guide
- Published: 2026-04-05

---

**To configure TurboQuant for the MLX-VLM server, pass a non-integer bit-width (such as `3.5`) via the `--kv-bits` flag or `KV_BITS` environment variable, and set the quantization scheme to `turboquant` using `--kv-quant-scheme` or `KV_QUANT_SCHEME` before starting the server.**

TurboQuant is a KV-cache quantization backend in the MLX-VLM repository that compresses keys and values during generation, enabling significantly longer prompts with reduced memory footprint. When you configure TurboQuant for the MLX-VLM server, the system automatically replaces standard cache objects with optimized `TurboQuantKVCache` instances that leverage custom metal kernels for packed data operations. This guide covers the specific configuration parameters, source code mechanics, and verification steps required to activate this backend.

## TurboQuant Configuration Requirements

Activating TurboQuant requires satisfying at least one of two conditions detected by the `turboquant_enabled()` function in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) (lines 58-64):

- **Non-integer bit-width**: Passing a fractional value like `3.5` automatically triggers TurboQuant, using the floor of the value for keys (3 bits) and the ceiling for values (4 bits).
- **Explicit scheme selection**: Setting `kv_quant_scheme` to `"turboquant"` forces the backend regardless of whether the bit-width is fractional.

The `_validate_bits()` function in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) (lines 46-55) validates that only integer or half-integer bit-widths (e.g., `2.0`, `3.5`, `4.0`) are accepted.

## Configuration Methods

You can configure TurboQuant through three interfaces: CLI arguments, environment variables, or programmatic Python.

### Command-Line Interface

Pass the quantization parameters directly to the `mlx_vlm server` entry point:

```bash
mlx_vlm server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --host 0.0.0.0 \
  --port 8080

```

The `get_quantized_kv_bits()` function in [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) (lines 52-61) parses the `--kv-bits` argument and returns `None` if set to `0` or if the model is QAT-aware. The `get_kv_quant_scheme()` function (lines 68-70) reads the `--kv-quant-scheme` flag.

### Environment Variables

For Docker containers or systemd services, export the configuration before launching:

```bash
export KV_BITS=3.5
export KV_QUANT_SCHEME=turboquant
export PRELOAD_MODEL=mlx-community/Qwen3.5-4B-4bit

mlx_vlm server --host 0.0.0.0 --port 8080

```

The server reads these via `os.environ.get` calls in [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) during initialization.

### Programmatic Python

Import the server module and pass arguments as a dictionary:

```python
from mlx_vlm import server

server_args = {
    "model": "mlx-community/Qwen3.5-4B-4bit",
    "kv_bits": 3.5,              # Non-integer triggers TurboQuant

    "kv_quant_scheme": "turboquant",
    "host": "0.0.0.0",
    "port": 8080,
}

server.main(**server_args)

```

This approach mirrors the CLI argument parsing and triggers the same quantization pipeline in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py).

## How the Configuration is Processed

Once the server initializes, the configuration flows through specific functions in the MLX-VLM codebase:

1. **Configuration parsing**: [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py) extracts `kv_bits` via `get_quantized_kv_bits()` and `kv_quant_scheme` via `get_kv_quant_scheme()` from either CLI flags or environment variables.
2. **Enablement check**: `turboquant_enabled()` in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) returns `True` if the bit-width is non-integer or the scheme equals `"turboquant"`.
3. **Cache conversion**: During generation, `maybe_quantize_kv_cache()` in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) (lines 42-86) traverses the `PromptCache` structure and replaces standard `KVCache` objects with `TurboQuantKVCache` instances once the generation offset passes `quantized_kv_start`.
4. **Quantized execution**: The `TurboQuantKVCache` class (defined around line 4790 in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py)) stores keys and values in a rotated-Hadamard-transformed format, allowing subsequent attention passes to read directly from packed representations without full de-quantization.

## Verifying TurboQuant Activation

To confirm that TurboQuant is active after configuration, inspect the prompt cache objects programmatically:

```python
from mlx_vlm.generate import maybe_quantize_kv_cache
from mlx_vlm.turboquant import TurboQuantKVCache

# After first generation call

prompt_cache = [...]  # Obtained from generation context

maybe_quantize_kv_cache(
    prompt_cache,
    quantized_kv_start=0,
    kv_group_size=64,
    kv_bits=3.5,
    kv_quant_scheme="turboquant"
)

# Verify conversion occurred

assert isinstance(prompt_cache[0], TurboQuantKVCache)
print("TurboQuant KV cache is active")

```

This verification confirms that `maybe_quantize_kv_cache()` successfully swapped the cache implementation according to your MLX-VLM server configuration.

## Recommended Settings

For production deployments, use **3.5-bit quantization** (3-bit keys, 4-bit values), which provides approximately 4.5× compression with negligible quality loss. This setting satisfies the `_validate_bits()` constraints and represents the optimal balance between memory reduction and generation accuracy.

## Summary

- Configure TurboQuant by setting `--kv-bits` to a non-integer value (e.g., `3.5`) and `--kv-quant-scheme` to `turboquant` when launching the MLX-VLM server.
- Alternative configuration methods include `KV_BITS` and `KV_QUANT_SCHEME` environment variables or passing arguments to `server.main()` in Python.
- The server processes these settings through `get_quantized_kv_bits()` and `get_kv_quant_scheme()` in [`mlx_vlm/server.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/server.py).
- TurboQuant activates when `turboquant_enabled()` detects valid parameters, triggering `maybe_quantize_kv_cache()` in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) to replace standard caches with `TurboQuantKVCache` objects.
- Use 3.5-bit width for optimal memory efficiency and inference performance.

## Frequently Asked Questions

### What bit-widths does TurboQuant support?

TurboQuant supports integer and half-integer bit-widths (such as `2.0`, `3.0`, `3.5`, `4.0`) as validated by the `_validate_bits()` function in [`mlx_vlm/turboquant.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/turboquant.py) (lines 46-55). The most common production setting is `3.5` bits, which allocates 3 bits to keys and 4 bits to values for balanced compression.

### Can I use TurboQuant with standard integer bit-widths?

Yes, you can use integer bit-widths like `4.0`, but you must explicitly set `--kv-quant-scheme turboquant` or `KV_QUANT_SCHEME=turboquant` to force the TurboQuant backend. Without the explicit scheme, non-integer values are the only automatic trigger for TurboQuant activation in `turboquant_enabled()`.

### How do I know if TurboQuant is actually running?

After starting the server and processing at least one generation request, check that the prompt cache contains `TurboQuantKVCache` instances rather than standard `KVCache` objects. The `maybe_quantize_kv_cache()` function in [`mlx_vlm/generate.py`](https://github.com/Blaizzy/mlx-vlm/blob/main/mlx_vlm/generate.py) performs this swap once the generation offset exceeds `quantized_kv_start`, typically immediately when set to `0`.

### Does TurboQuant work with all model architectures?

TurboQuant works with standard transformer architectures supported by MLX-VLM, but it skips rotating caches or array-based caches during the conversion process in `maybe_quantize_kv_cache()`. The quantization applies to regular `KVCache` objects used by most multimodal language models in the repository.