What Is CQ2-Bit Quantization and How Is It Used in Needle 2?

CQ2-bit quantization is a learned codebook compression technique that packs 45 million parameters into 14 MB by representing each weight with only 2 bits, utilizing Hadamard rotation to preserve model quality.

CQ2-bit quantization serves as the flagship compression engine in the Needle 2 framework, enabling deployment of large language models on memory-constrained devices without sacrificing accuracy. As part of the Cactus Quants family, this method reduces weight precision to 2 bits per value through vector quantization and orthogonal transformation. The complete implementation resides in needle/model/quantize.py and supports both uniform and mixed-precision quantization strategies.

How CQ2-Bit Quantization Works

The CQ2-bit algorithm compresses neural network weights through a three-stage pipeline that decorrelates dimensions before applying scalar quantization.

Hadamard Rotation and Decorrelation

First, the algorithm applies a fixed Walsh-Hadamard matrix transform to weight groups. In needle/model/quantize.py, the cq_quantize function (starting at line 41) generates the rotation matrix via _cq_hadamard_np(g) and applies it through matrix multiplication:

H = jnp.asarray(_cq_hadamard_np(g))
rot = groups @ H

This rotation decorrelates the weight dimensions, transforming the distribution into one more amenable to simple scalar quantization.

Codebook Generation and Quantization

Following rotation, the algorithm scales each group to match a learned Lloyd-Max codebook. The _cq_codebook_np function at line 13 generates normalized codebook vectors for the target bit-width. The quantization pipeline at lines 44-46 performs normalization, nearest-neighbor lookup, and de-rotation:

unit = rot / jnp.maximum(norm, 1e-12)
quant = _cq_nearest(unit, cb)
deq = (quant * norm) @ H

This process maps continuous values to discrete 2-bit indices while preserving the angular relationships critical for model performance.

Implementation Details in needle/model/quantize.py

The quantization system supports multiple bit-widths through a flexible API centered around three core functions.

Supported Bit Widths

The constant CQ_BITS defined at line 199 specifies the available quantization levels:

CQ_BITS = (2, 3, 4)

While the system supports 2, 3, and 4-bit modes, CQ2-bit remains the default configuration for Needle 2 deployments, offering the optimal balance between compression ratio and accuracy.

Mixed-Precision Deployment

For heterogeneous model architectures, cq_mixed_params (line 21) enables per-layer bit-width selection through a mapping dictionary:

def cq_mixed_params(params, names, bits_map, default_bits=2):
    b = _bits_for(names[i], bits_map, default_bits)
    # ... applies varying precision per layer

This allows sensitive layers to retain higher precision while compressing robust layers to the standard 2-bit format.

Deployment Helper

The deploy_quantize function at line 74 returns a fully quantized parameter tree ready for inference:

def deploy_quantize(params, config):
    return cq_quantize_params(params, _WEIGHT_BITS, _WEIGHT_GROUP)

By default, _WEIGHT_BITS initializes to 2, enforcing the CQ2-bit format unless explicitly overridden.

Deploying CQ2-Bit Quantized Models

The following workflow demonstrates how to apply CQ2-bit quantization to a Needle 2 checkpoint and export it to a portable binary:

import needle
from needle.model.quantize import configure_deploy, deploy_quantize

# Load a pretrained checkpoint (weights are downloaded automatically)

params = needle.model.load_weights("cactus-compute/needle2")

# Configure deployment for CQ2-bit (2-bit weights, 8-bit activations)

configure_deploy(act_bits=8, kv_bits=0)   # kv_bits=0 disables KV cache quantization

# Quantize the checkpoint for deployment

quantized_params, spec = deploy_quantize(params, config=None)
print(f"Deployment spec: {spec}")  # → "CQ W2"

# Export to a portable .cact archive

needle.build(
    checkpoint=params,
    out="my_needle.cact",
    quantized_params=quantized_params,
    bits=2                     # Forces CQ2-bit encoding

)

The resulting my_needle.cact file contains all quantized parameters in a single 14 MB binary that the Needle 2 runtime can load directly into approximately 28 MB of RAM, eliminating the need for external model files.

Summary

• CQ2-bit quantization compresses 45M-parameter models to 14 MB by representing weights with 2 bits per value through learned codebook compression.
• Hadamard rotation in cq_quantize decorrelates weight dimensions before quantization, preserving model accuracy despite aggressive compression.
• The implementation supports 2, 3, and 4-bit modes via the CQ_BITS tuple, with 2-bit as the default for maximum compression.
• Mixed-precision deployment via cq_mixed_params allows selective quantization of specific layers based on sensitivity.
• Single-file deployment produces .cact archives that contain all necessary weights for edge inference without additional dependencies.

Frequently Asked Questions

What compression ratio does CQ2-bit quantization achieve?

CQ2-bit quantization achieves approximately a 16:1 compression ratio compared to 32-bit floating point, packing a 45 million parameter model into roughly 14 MB. When loaded for inference, the model consumes approximately 28 MB of RAM, including overhead for the codebook and runtime buffers.

How does Hadamard rotation improve quantization accuracy?

Hadamard rotation multiplies weight groups by a fixed Walsh-Hadamard matrix before quantization, decorrelating the dimensions and flattening the distribution. This preprocessing step allows a simple scalar quantizer to perform effectively on the rotated space, minimizing the angular error introduced by aggressive 2-bit discretization.

Can I use bit widths other than 2 bits for quantization?

Yes. While CQ2-bit represents the default configuration, the system supports 3-bit and 4-bit quantization through the CQ_BITS = (2, 3, 4) constant. You can specify alternative bit widths by passing the bits parameter to needle.build() or configuring _WEIGHT_BITS directly before calling deploy_quantize.

How do I export a model with CQ2-bit quantization for production?

Export a CQ2-bit quantized model by calling deploy_quantize() to generate quantized parameters, then pass these to needle.build() with bits=2. This produces a standalone .cact file containing all quantized weights and metadata required by the Needle 2 inference engine, suitable for deployment on resource-constrained devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}