Cactus Quants CQ2-bit Quantization: Extreme Compression for Edge LLMs

Cactus Quants CQ2-bit quantization is a custom 2-bit weight compression scheme in Needle 2 that uses Hadamard rotation and group-wise codebooks to shrink models to 14 MB while preserving inference accuracy.

Needle 2 achieves aggressive model compression through Cactus Quants CQ2-bit quantization, a proprietary method that packs 45 million parameters into just 2 bits per weight. This article examines the implementation in needle/model/quantize.py to explain how orthonormal transforms and Lloyd-Max codebooks make extreme quantization feasible for resource-constrained devices.

How CQ2-bit Quantization Works

The CQ2-bit system reduces precision through a pipeline of linear algebra operations and lookup tables designed to minimize reconstruction error.

Group-wise Weight Partitioning

Weights are divided into fixed-size groups defaulting to 128 elements (_WEIGHT_GROUP), with each group quantized independently. This strategy contains error propagation within local neighborhoods and enables hardware-friendly parallel operations. The partitioning logic resides in fake_quant and the entry-point function cq_quantize, which prepare tensors for the compression pipeline.

Hadamard Rotation

Before quantization, each weight group undergoes rotation by an orthonormal Walsh-Hadamard matrix via _cq_hadamard_np. This transformation spreads information uniformly across dimensions, eliminating axis-aligned outliers that would otherwise dominate scalar quantization error. The rotation step is critical for maintaining representational power when reducing precision to 2 bits.

Optimized Codebook Design

Cactus Quants employs two specialized codebook strategies:

• _TERNARY_CB – A ternary codebook supporting the special "1.58-bit" ternary quantization case.
• _lloyd_max_gaussian – An optimal Lloyd-Max quantizer for Gaussian-distributed weights, used for 2-bit and other bit-widths.

The selected codebook is scaled by group size and cached for reuse during both training and inference.

The CQ2-bit Implementation Pipeline

The core quantization routine cq_quantize (lines 34-48 of needle/model/quantize.py) executes a deterministic sequence:

1. Pad and reshape weight tensors into group-aligned matrices.
2. Rotate using the Hadamard transform to decorrelate magnitudes.
3. Normalize the rotated values to match the codebook dynamic range.
4. Lookup nearest codebook entries using vectorized operations.
5. Inverse rotate to restore the original orientation of weight groups.
6. Unpad to return tensors to their original shape.

The output is a de-quantized tensor that approximates the original weights using only 2 bits per stored value, plus a small per-group overhead for scaling factors.

Model Integration and Deployment

The quantization system exposes high-level APIs that wrap the low-level pipeline for seamless integration with JAX-based training workflows.

Configuration and Export

The configure_deploy function (lines 61-68) sets global quantization hyperparameters—including the target bit-width (_WEIGHT_BITS) and group size—while clearing JAX compilation caches to ensure the new configuration takes effect. During export, deploy_quantize (lines 74-81) traverses the model parameter tree, applying cq_quantize to every eligible weight leaf and generating a specification string (e.g., "CQ W2") that identifies the compression format.

Training with Quantization

For fine-tuning already-compressed models, cq_ste_params (lines 58-60) implements a Straight-Through Estimator (STE). This technique retains the full-precision tensor in the computational graph for gradient computation while using the 2-bit quantized version for forward passes, enabling quantization-aware training (QAT) without destabilizing optimization.

Practical Implementation Example

The following example demonstrates configuring a Needle 2 model for CQ2-bit deployment and applying the compression pipeline:

import needle
from needle.model import quantize as q

# Load model and inspect default quantization settings

model = needle.Needle()
print(q._WEIGHT_GROUP)  # Output: 128

# Configure for deployment with CQ2-bit weights

q.configure_deploy(act_bits=8, kv_bits=8, kv_group=64)

# Apply quantization to all parameters

params = model.params
quantized_params, spec = q.deploy_quantize(params, config=model)
print(spec)  # Output: "CQ W2"

# Fine-tune using straight-through estimator

quantized_params = q.cq_ste_params(params, bits=2)

Why 2-bit Compression Matters

By compressing every weight to 2 bits, Needle 2 reduces its 45-million-parameter model to a 14 MB binary, enabling inference on devices with as little as 28 MB of RAM. This represents an 8× size reduction compared to standard FP16 weights and a 2× reduction over 4-bit alternatives, making large language models viable for microcontrollers and edge hardware without requiring external memory.

Summary

• Cactus Quants CQ2-bit quantization reduces Needle 2 model weights to 2 bits per value using group-wise processing and Hadamard rotation.
• The implementation in needle/model/quantize.py uses Lloyd-Max Gaussian codebooks for optimal 2-bit reconstruction.
• The pipeline includes pad, rotate, normalize, lookup, inverse rotate, and unpad stages to minimize quantization error.
• deploy_quantize handles model export, while cq_ste_params enables quantization-aware fine-tuning via straight-through estimation.
• The compression allows a 45M parameter model to fit in 14 MB, suitable for 28 MB RAM devices.

Frequently Asked Questions

What distinguishes CQ2-bit from standard INT2 quantization?

CQ2-bit differs from naive INT2 schemes by applying a Hadamard rotation before quantization, which reduces the dynamic range of weight distributions and improves reconstruction accuracy. Unlike symmetric INT2, it uses learned Lloyd-Max codebooks optimized for Gaussian weight distributions.

Why is Hadamard rotation applied before quantizing?

The Walsh-Hadamard transform spreads information entropy evenly across all dimensions of a weight group, preventing a few large-magnitude outliers from consuming the limited representational capacity of 2-bit codes. This rotation, implemented in _cq_hadamard_np, is essential for maintaining model quality at extreme compression ratios.

How does group size affect CQ2-bit quantization?

The default group size of 128 (_WEIGHT_GROUP) balances memory overhead against quantization error. Smaller groups allow finer-grained scaling factors but increase storage overhead, while larger groups improve compression efficiency at the cost of potentially higher approximation error within each group.

Can I fine-tune a model after applying CQ2-bit quantization?

Yes. The cq_ste_params function implements straight-through estimation, allowing gradients to flow through the quantization layer during backpropagation while keeping weights at 2-bit precision for forward computation. This enables continued fine-tuning of CQ2-bit compressed models without full-precision storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}