How Cactus Quants Achieves CQ2-Bit Compression for Needle 2

Cactus Quants achieves CQ2-bit compression by applying a block-wise Hadamard rotation, per-group L2 normalization, and Lloyd-Max optimized scalar quantization to compress each weight to one of four levels, then packing the result for fast inference.

Cactus Quants implements Compressed-Quant (CQ) as a deterministic, block-wise quantizer designed for low-bit inference in the Needle 2 framework. The CQ2-bit mode squeezes model weights to just two bits per element while preserving accuracy through a tight integration of optimal codebooks, orthogonal transforms, and per-group normalization. According to the cactus-compute/needle source code, the entire pipeline lives in needle/model/quantize.py and is orchestrated by a small set of core functions.

Core CQ2-Bit Compression Pipeline in needle/model/quantize.py

The CQ quantizer processes weight matrices in three mathematical steps: codebook generation, Hadamard rotation, and scalar quantization with inverse rotation. A fourth step, cq_quantize_params, applies this block-wise routine across the model tree only to kernel and embedding tensors.

Step 1 – Codebook Generation with _cq_codebook_np

For a requested bit-width b, the quantizer builds a static codebook of optimal scalar levels. When bits = 2, the function _cq_codebook_np(bits, group_size) returns a codebook containing four normalized levels derived from a Lloyd-Max Gaussian optimizer (_lloyd_max_gaussian) or, in ternary mode, from the predefined _TERNARY_CB constant. These four levels represent the only values a quantized weight can take in CQ2-bit mode.

Step 2 – Hadamard Rotation via _cq_hadamard_np

Each weight matrix is split into column groups of size group_size = 128. The helper _cq_hadamard_np supplies a Hadamard matrix H, and cq_quantize performs the rotation rot = groups @ H. This orthogonal transform spreads information uniformly across dimensions, decorrelating the columns so that aggressive scalar quantization introduces less structured error.

Step 3 – Scalar Quantization and Inverse Rotation

After rotation, each group vector is L2-normalized to produce a unit vector and a per-group scale (norm). The helper _cq_nearest(unit, cb) snaps every component to the nearest entry in the 2-bit codebook. The de-quantized approximation is then recovered with inverse rotation:

deq = (_cq_nearest(unit, cb) * norm) @ H

The final tensor is sliced back to its original shape, preserving the approximate weight values at two bits of storage per element.

Step 4 – Block-Level Model Quantization with cq_quantize_params

The top-level function cq_quantize_params traverses the model parameter tree and applies the above routine selectively to kernel and embedding tensors. For kernels where the last two axes are transposed according to the _reduces_second_last predicate, the function swaps axes before quantization and restores the original layout afterward. This ensures the compressed representation remains compatible with the forward pass.

Why CQ2-Bit Quantization Retains High Fidelity

The CQ2-bit mode works well because of three specific design choices implemented in cactus-compute/needle:

• Hadamard rotation (_cq_hadamard_np) decorrelates matrix columns before quantization, enabling a tiny four-level codebook to capture most of the variance.
• Per-group normalization preserves energy locally, limiting quantization error to a small scale factor per 128-element block.
• Group size of 128 aligns with GPU and CPU memory boundaries and allows the Hadamard transform to be computed efficiently without excessive overhead.

Together these ingredients give a compact CQ2-bit representation that can be packed and streamed to the inference engine with minimal runtime cost.

Practical Code Examples

The following examples show how to quantize a trained model to CQ2-bit and prepare it for deployment.

Quantizing Weights to CQ2-Bit

import jax.numpy as jnp
from needle.model.quantize import cq_quantize_params, CQ_BITS

# params is a nested dict of JAX arrays from a trained Needle model

bits = 2          # CQ_BITS[0]

group_sz = 128    # default block size used by Cactus Quants

# Perform block-wise CQ quantization

quantized_params = cq_quantize_params(params, bits, group_sz)

Deploying a Model with CQ2-Bit Weights

from needle.model.export import export
from needle.model.quantize import deploy_quantize

# config may specify weight_bits, e.g., config.weight_bits = "default=2"

quant_params, spec = deploy_quantize(params, config)  # CQ2-bit internally

export(quant_params, spec, out_path="model.cq2")

Key Source Files for CQ2-Bit Compression

Understanding the full CQ2-bit stack requires looking at three files in the cactus-compute/needle repository:

• needle/model/quantize.py – Contains _cq_codebook_np, _cq_hadamard_np, _cq_nearest, cq_quantize, and cq_quantize_params. This is the core of the CQ2-bit implementation.
• needle/model/export.py – Packs CQ tensors into a binary format for runtime loading and inference.
• needle/model/architecture.py – Connects the quantized weights to the forward pass via the quant flag.

The constant CQ_BITS = (2, 3, 4) at the bottom of quantize.py explicitly declares the supported bit-widths, with 2 being the most aggressive compression tier.

Summary

• Cactus Quants uses a block-wise orthogonal transform pipeline in needle/model/quantize.py to produce CQ2-bit weights.
• _cq_codebook_np generates a four-level codebook for 2-bit mode using Lloyd-Max optimization or a predefined ternary table.
• _cq_hadamard_np rotates 128-column groups to decorrelate weights before scalar quantization.
• _cq_nearest maps normalized unit vectors to the codebook, and inverse rotation restores approximate weights.
• cq_quantize_params applies the process across the model tree, handling transposed kernels automatically.
• The final packed format is written by needle/model/export.py for efficient runtime inference.

Frequently Asked Questions

What is CQ2-bit compression in Needle 2?

CQ2-bit compression is a mode of the Compressed-Quant (CQ) system that stores each model weight using only two bits. It is implemented in cactus-compute/needle as a deterministic pipeline combining Hadamard rotation, per-group normalization, and four-level scalar quantization.

How does the Hadamard rotation improve 2-bit quantization?

The Hadamard rotation, performed by _cq_hadamard_np and applied in cq_quantize as groups @ H, spreads weight information uniformly across dimensions. This decorrelation step prevents the tiny 2-bit codebook from missing critical variance, which dramatically reduces reconstruction error.

What group size does Cactus Quants use for CQ2-bit mode?

The default and recommended group size is 128 columns. This block size is chosen for efficient memory alignment on GPUs and CPUs and for fast computation of the Hadamard transform inside needle/model/quantize.py.

Where are CQ2-bit weights packed for runtime inference?

After quantization, deploy_quantize and needle/model/export.py pack the CQ2-bit tensors into a binary stream. The resulting file can be loaded by the inference engine with the quant=True path in needle/model/architecture.py, keeping dequantization overhead minimal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}