# Cactus Quants CQ2-bit Quantization: Extreme Compression for Edge LLMs

> Discover Cactus Quants CQ2-bit quantization a custom 2-bit weight compression scheme that shrinks LLMs to 14 MB using Hadamard rotation and group-wise codebooks while maintaining accuracy.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-05

---

**Cactus Quants CQ2-bit quantization is a custom 2-bit weight compression scheme in Needle 2 that uses Hadamard rotation and group-wise codebooks to shrink models to 14 MB while preserving inference accuracy.**

Needle 2 achieves aggressive model compression through **Cactus Quants CQ2-bit quantization**, a proprietary method that packs 45 million parameters into just 2 bits per weight. This article examines the implementation in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) to explain how orthonormal transforms and Lloyd-Max codebooks make extreme quantization feasible for resource-constrained devices.

## How CQ2-bit Quantization Works

The CQ2-bit system reduces precision through a pipeline of linear algebra operations and lookup tables designed to minimize reconstruction error.

### Group-wise Weight Partitioning

Weights are divided into fixed-size groups defaulting to **128 elements** (`_WEIGHT_GROUP`), with each group quantized independently. This strategy contains error propagation within local neighborhoods and enables hardware-friendly parallel operations. The partitioning logic resides in `fake_quant` and the entry-point function `cq_quantize`, which prepare tensors for the compression pipeline.

### Hadamard Rotation

Before quantization, each weight group undergoes rotation by an orthonormal Walsh-Hadamard matrix via `_cq_hadamard_np`. This transformation spreads information uniformly across dimensions, eliminating axis-aligned outliers that would otherwise dominate scalar quantization error. The rotation step is critical for maintaining representational power when reducing precision to 2 bits.

### Optimized Codebook Design

Cactus Quants employs two specialized codebook strategies:

- **`_TERNARY_CB`** – A ternary codebook supporting the special "1.58-bit" ternary quantization case.
- **`_lloyd_max_gaussian`** – An optimal Lloyd-Max quantizer for Gaussian-distributed weights, used for 2-bit and other bit-widths.

The selected codebook is scaled by group size and cached for reuse during both training and inference.

## The CQ2-bit Implementation Pipeline

The core quantization routine `cq_quantize` (lines 34-48 of [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)) executes a deterministic sequence:

1. **Pad and reshape** weight tensors into group-aligned matrices.
2. **Rotate** using the Hadamard transform to decorrelate magnitudes.
3. **Normalize** the rotated values to match the codebook dynamic range.
4. **Lookup** nearest codebook entries using vectorized operations.
5. **Inverse rotate** to restore the original orientation of weight groups.
6. **Unpad** to return tensors to their original shape.

The output is a de-quantized tensor that approximates the original weights using only 2 bits per stored value, plus a small per-group overhead for scaling factors.

## Model Integration and Deployment

The quantization system exposes high-level APIs that wrap the low-level pipeline for seamless integration with JAX-based training workflows.

### Configuration and Export

The `configure_deploy` function (lines 61-68) sets global quantization hyperparameters—including the target bit-width (`_WEIGHT_BITS`) and group size—while clearing JAX compilation caches to ensure the new configuration takes effect. During export, `deploy_quantize` (lines 74-81) traverses the model parameter tree, applying `cq_quantize` to every eligible weight leaf and generating a specification string (e.g., `"CQ W2"`) that identifies the compression format.

### Training with Quantization

For fine-tuning already-compressed models, `cq_ste_params` (lines 58-60) implements a **Straight-Through Estimator (STE)**. This technique retains the full-precision tensor in the computational graph for gradient computation while using the 2-bit quantized version for forward passes, enabling quantization-aware training (QAT) without destabilizing optimization.

## Practical Implementation Example

The following example demonstrates configuring a Needle 2 model for CQ2-bit deployment and applying the compression pipeline:

```python
import needle
from needle.model import quantize as q

# Load model and inspect default quantization settings

model = needle.Needle()
print(q._WEIGHT_GROUP)  # Output: 128

# Configure for deployment with CQ2-bit weights

q.configure_deploy(act_bits=8, kv_bits=8, kv_group=64)

# Apply quantization to all parameters

params = model.params
quantized_params, spec = q.deploy_quantize(params, config=model)
print(spec)  # Output: "CQ W2"

# Fine-tune using straight-through estimator

quantized_params = q.cq_ste_params(params, bits=2)

```

## Why 2-bit Compression Matters

By compressing every weight to 2 bits, Needle 2 reduces its 45-million-parameter model to a **14 MB binary**, enabling inference on devices with as little as **28 MB of RAM**. This represents an 8× size reduction compared to standard FP16 weights and a 2× reduction over 4-bit alternatives, making large language models viable for microcontrollers and edge hardware without requiring external memory.

## Summary

- **Cactus Quants CQ2-bit quantization** reduces Needle 2 model weights to 2 bits per value using group-wise processing and Hadamard rotation.
- The implementation in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) uses Lloyd-Max Gaussian codebooks for optimal 2-bit reconstruction.
- The pipeline includes pad, rotate, normalize, lookup, inverse rotate, and unpad stages to minimize quantization error.
- **`deploy_quantize`** handles model export, while **`cq_ste_params`** enables quantization-aware fine-tuning via straight-through estimation.
- The compression allows a 45M parameter model to fit in 14 MB, suitable for 28 MB RAM devices.

## Frequently Asked Questions

### What distinguishes CQ2-bit from standard INT2 quantization?

CQ2-bit differs from naive INT2 schemes by applying a Hadamard rotation before quantization, which reduces the dynamic range of weight distributions and improves reconstruction accuracy. Unlike symmetric INT2, it uses learned Lloyd-Max codebooks optimized for Gaussian weight distributions.

### Why is Hadamard rotation applied before quantizing?

The Walsh-Hadamard transform spreads information entropy evenly across all dimensions of a weight group, preventing a few large-magnitude outliers from consuming the limited representational capacity of 2-bit codes. This rotation, implemented in `_cq_hadamard_np`, is essential for maintaining model quality at extreme compression ratios.

### How does group size affect CQ2-bit quantization?

The default group size of 128 (`_WEIGHT_GROUP`) balances memory overhead against quantization error. Smaller groups allow finer-grained scaling factors but increase storage overhead, while larger groups improve compression efficiency at the cost of potentially higher approximation error within each group.

### Can I fine-tune a model after applying CQ2-bit quantization?

Yes. The `cq_ste_params` function implements straight-through estimation, allowing gradients to flow through the quantization layer during backpropagation while keeping weights at 2-bit precision for forward computation. This enables continued fine-tuning of CQ2-bit compressed models without full-precision storage.