Supported Bit Widths for TurboQuant in MLX-VLM: Integer and Half-Integer Values Explained

TurboQuant supports bit widths that are integers (1, 2, 3...) or half-integers (1.5, 2.5, 3.5...) with a minimum value of 1, rejecting all other fractional values with a ValueError.

TurboQuant is a quantization method implemented in the mlx-vlm repository for efficient vision-language model inference. When configuring key-value cache quantization, developers must specify valid kv_bits parameters that align with the codec's precision requirements. The implementation explicitly validates these values at instantiation to ensure compatibility with the underlying quantization kernels.

Allowed Bit Width Values for TurboQuant

TurboQuant accepts two distinct categories of bit widths, both requiring a minimum value of 1:

• Integer values: Whole numbers such as 1, 2, 3, 4, and higher
• Half-integer values: Numbers ending in .5 such as 3.5, 4.5, 5.5, providing finer granularity between whole-bit precisions

Any value below 1 or containing fractional components other than .5 will trigger validation errors. For example, 3.0, 4, and 5.5 are valid configurations, while 0.5 or 3.25 are rejected.

Implementation Details in MLX-VLM Source Code

The validation logic resides in the mlx_vlm/turboquant.py module, where the constructor performs strict type checking before allowing instantiation.

Constructor Validation in turboquant.py

At line 3453 of mlx_vlm/turboquant.py, the implementation validates the kv_bits parameter and raises a descriptive error for unsupported values:

raise ValueError("TurboQuant currently supports integer and .5 bit-widths, got {bits}.")

This check ensures that the codec receives only precision values it can encode efficiently. The validation occurs immediately upon creating a TurboQuant instance, preventing runtime errors during the quantization process.

Unit Test Coverage

The test suite in mlx_vlm/tests/test_turboquant.py confirms both integer and half-integer support through explicit test cases at lines 79 and 98:

# Half-integer bit width

kv_bits=3.5

# Integer bit width  

kv_bits=3.0

These tests verify that the quantization pipeline accepts both 3.5 (half-integer) and 3.0 (integer) without raising exceptions, validating the core functionality across different precision levels.

Practical Usage Examples

When initializing TurboQuant for KV cache quantization, use only supported values:

from mlx_vlm.turboquant import TurboQuant

# Valid configurations

quant_4bit = TurboQuant(kv_bits=4)      # Integer

quant_3_5bit = TurboQuant(kv_bits=3.5)  # Half-integer

quant_2bit = TurboQuant(kv_bits=2.0)    # Integer as float

# Invalid - will raise ValueError

try:
    invalid_quant = TurboQuant(kv_bits=3.25)
except ValueError as e:
    print(e)  # TurboQuant currently supports integer and .5 bit-widths, got 3.25.

Summary

• TurboQuant requires kv_bits values to be integers or half-integers (integer + 0.5)
• The minimum supported value is 1; anything lower raises a ValueError
• Valid examples include 1, 2.5, 3, 4.5, 8, etc.
• Invalid values like 3.25 or 0.5 trigger explicit validation errors in mlx_vlm/turboquant.py
• Unit tests in test_turboquant.py verify both integer and half-integer code paths

Frequently Asked Questions

Does TurboQuant support 4-bit quantization?

Yes, kv_bits=4 is fully supported as a valid integer bit width. This is one of the most common configurations for balancing memory efficiency and model accuracy in vision-language models.

What happens if I try to use 3.25 bits?

The constructor raises a ValueError with the message: "TurboQuant currently supports integer and .5 bit-widths, got 3.25." This strict validation prevents the codec from attempting to quantize to unsupported precision levels that lack kernel implementations.

Is there a performance difference between integer and half-integer bit widths?

While both types are supported, half-integer values like 3.5 provide finer granularity for memory-accuracy tradeoffs. The performance characteristics depend on the specific hardware acceleration available in MLX, but both formats undergo the same validation pipeline in mlx_vlm/turboquant.py.

What is the minimum supported bit width for TurboQuant?

The minimum value is 1 bit. Attempting to instantiate TurboQuant with kv_bits=0.5 or any value less than 1 will raise a ValueError: TurboQuant requires kv_bits >= 1, as enforced by the validation logic in the source code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}