How to Configure TurboQuant for the MLX-VLM Server: Complete Setup Guide

To configure TurboQuant for the MLX-VLM server, pass a non-integer bit-width (such as 3.5) via the --kv-bits flag or KV_BITS environment variable, and set the quantization scheme to turboquant using --kv-quant-scheme or KV_QUANT_SCHEME before starting the server.

TurboQuant is a KV-cache quantization backend in the MLX-VLM repository that compresses keys and values during generation, enabling significantly longer prompts with reduced memory footprint. When you configure TurboQuant for the MLX-VLM server, the system automatically replaces standard cache objects with optimized TurboQuantKVCache instances that leverage custom metal kernels for packed data operations. This guide covers the specific configuration parameters, source code mechanics, and verification steps required to activate this backend.

TurboQuant Configuration Requirements

Activating TurboQuant requires satisfying at least one of two conditions detected by the turboquant_enabled() function in mlx_vlm/turboquant.py (lines 58-64):

• Non-integer bit-width: Passing a fractional value like 3.5 automatically triggers TurboQuant, using the floor of the value for keys (3 bits) and the ceiling for values (4 bits).
• Explicit scheme selection: Setting kv_quant_scheme to "turboquant" forces the backend regardless of whether the bit-width is fractional.

The _validate_bits() function in mlx_vlm/turboquant.py (lines 46-55) validates that only integer or half-integer bit-widths (e.g., 2.0, 3.5, 4.0) are accepted.

Configuration Methods

You can configure TurboQuant through three interfaces: CLI arguments, environment variables, or programmatic Python.

Command-Line Interface

Pass the quantization parameters directly to the mlx_vlm server entry point:

mlx_vlm server \
  --model mlx-community/Qwen3.5-4B-4bit \
  --kv-bits 3.5 \
  --kv-quant-scheme turboquant \
  --host 0.0.0.0 \
  --port 8080

The get_quantized_kv_bits() function in mlx_vlm/server.py (lines 52-61) parses the --kv-bits argument and returns None if set to 0 or if the model is QAT-aware. The get_kv_quant_scheme() function (lines 68-70) reads the --kv-quant-scheme flag.

Environment Variables

For Docker containers or systemd services, export the configuration before launching:

export KV_BITS=3.5
export KV_QUANT_SCHEME=turboquant
export PRELOAD_MODEL=mlx-community/Qwen3.5-4B-4bit

mlx_vlm server --host 0.0.0.0 --port 8080

The server reads these via os.environ.get calls in mlx_vlm/server.py during initialization.

Programmatic Python

Import the server module and pass arguments as a dictionary:

from mlx_vlm import server

server_args = {
    "model": "mlx-community/Qwen3.5-4B-4bit",
    "kv_bits": 3.5,              # Non-integer triggers TurboQuant

    "kv_quant_scheme": "turboquant",
    "host": "0.0.0.0",
    "port": 8080,
}

server.main(**server_args)

This approach mirrors the CLI argument parsing and triggers the same quantization pipeline in mlx_vlm/generate.py.

How the Configuration is Processed

Once the server initializes, the configuration flows through specific functions in the MLX-VLM codebase:

1. Configuration parsing: mlx_vlm/server.py extracts kv_bits via get_quantized_kv_bits() and kv_quant_scheme via get_kv_quant_scheme() from either CLI flags or environment variables.
2. Enablement check: turboquant_enabled() in mlx_vlm/turboquant.py returns True if the bit-width is non-integer or the scheme equals "turboquant".
3. Cache conversion: During generation, maybe_quantize_kv_cache() in mlx_vlm/generate.py (lines 42-86) traverses the PromptCache structure and replaces standard KVCache objects with TurboQuantKVCache instances once the generation offset passes quantized_kv_start.
4. Quantized execution: The TurboQuantKVCache class (defined around line 4790 in mlx_vlm/turboquant.py) stores keys and values in a rotated-Hadamard-transformed format, allowing subsequent attention passes to read directly from packed representations without full de-quantization.

Verifying TurboQuant Activation

To confirm that TurboQuant is active after configuration, inspect the prompt cache objects programmatically:

from mlx_vlm.generate import maybe_quantize_kv_cache
from mlx_vlm.turboquant import TurboQuantKVCache

# After first generation call

prompt_cache = [...]  # Obtained from generation context

maybe_quantize_kv_cache(
    prompt_cache,
    quantized_kv_start=0,
    kv_group_size=64,
    kv_bits=3.5,
    kv_quant_scheme="turboquant"
)

# Verify conversion occurred

assert isinstance(prompt_cache[0], TurboQuantKVCache)
print("TurboQuant KV cache is active")

This verification confirms that maybe_quantize_kv_cache() successfully swapped the cache implementation according to your MLX-VLM server configuration.

Recommended Settings

For production deployments, use 3.5-bit quantization (3-bit keys, 4-bit values), which provides approximately 4.5× compression with negligible quality loss. This setting satisfies the _validate_bits() constraints and represents the optimal balance between memory reduction and generation accuracy.

Summary

• Configure TurboQuant by setting --kv-bits to a non-integer value (e.g., 3.5) and --kv-quant-scheme to turboquant when launching the MLX-VLM server.
• Alternative configuration methods include KV_BITS and KV_QUANT_SCHEME environment variables or passing arguments to server.main() in Python.
• The server processes these settings through get_quantized_kv_bits() and get_kv_quant_scheme() in mlx_vlm/server.py.
• TurboQuant activates when turboquant_enabled() detects valid parameters, triggering maybe_quantize_kv_cache() in mlx_vlm/generate.py to replace standard caches with TurboQuantKVCache objects.
• Use 3.5-bit width for optimal memory efficiency and inference performance.

Frequently Asked Questions

What bit-widths does TurboQuant support?

TurboQuant supports integer and half-integer bit-widths (such as 2.0, 3.0, 3.5, 4.0) as validated by the _validate_bits() function in mlx_vlm/turboquant.py (lines 46-55). The most common production setting is 3.5 bits, which allocates 3 bits to keys and 4 bits to values for balanced compression.

Can I use TurboQuant with standard integer bit-widths?

Yes, you can use integer bit-widths like 4.0, but you must explicitly set --kv-quant-scheme turboquant or KV_QUANT_SCHEME=turboquant to force the TurboQuant backend. Without the explicit scheme, non-integer values are the only automatic trigger for TurboQuant activation in turboquant_enabled().

How do I know if TurboQuant is actually running?

After starting the server and processing at least one generation request, check that the prompt cache contains TurboQuantKVCache instances rather than standard KVCache objects. The maybe_quantize_kv_cache() function in mlx_vlm/generate.py performs this swap once the generation offset exceeds quantized_kv_start, typically immediately when set to 0.

Does TurboQuant work with all model architectures?

TurboQuant works with standard transformer architectures supported by MLX-VLM, but it skips rotating caches or array-based caches during the conversion process in maybe_quantize_kv_cache(). The quantization applies to regular KVCache objects used by most multimodal language models in the repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}