# What Is Cactus Quants (CQ) Quantization and Its Benefits

> Discover Cactus Quants (CQ) quantization a novel pipeline that drastically compresses neural networks to 2-bit precision. Achieve massive size reductions and efficient RAM usage for models like Needle 2.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-09

---

**Cactus Quants (CQ) is a bespoke quantization pipeline that compresses neural network weights to as low as 2-bit precision using Hadamard transforms, group-wise Lloyd-Max codebooks, and training-aware fine-tuning, enabling the 45M-parameter Needle 2 model to ship as a 14MB binary and run entirely in 28MB of RAM.**

Cactus Quants (CQ) is the proprietary compression engine developed by cactus-compute for the Needle repository. It powers the extreme efficiency of Needle 2 by applying advanced linear algebra and learned quantization tables to reduce model size by up to 25× compared to full-precision floats. The following sections break down the architecture of CQ, its implementation across [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) and [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), and the concrete performance gains it delivers on edge devices.

## Core Mechanics of CQ Quantization

### Group-Wise Quantization with Hadamard Rotation

At the heart of CQ is **group-wise quantization** that processes weights in fixed-size blocks (default **group_size** of 128) rather than tensor-wide scales. Before quantization, each block is rotated by a pre-computed Walsh-**Hadamard matrix** (`_cq_hadamard_np`). This transform spreads information uniformly, Gaussianizing the distribution and minimizing quantization error for low-bit representations. The rotation and scaling logic resides in **`cq_quantize`** (approximately lines 34–48 in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)), while the inverse operation reconstructs weights on-the-fly during inference.

### Learned Lloyd-Max Codebooks

CQ does not use uniform linear quantization. Instead, it generates **Lloyd-Max derived codebooks** (`_cq_codebook_np`, lines 13–16) that contain optimal centroids for each target bit-width (e.g., 2-bit or 4-bit). These codebooks are computed once per bit-width and shared across all compatible layers, ensuring that the most frequent weight values retain the highest fidelity while outliers are mapped to the nearest centroid.

### Mixed-Precision Bit Mapping

To preserve accuracy in critical layers while aggressively compressing others, CQ supports **per-layer bit-width configuration**. The function **`parse_bits_map`** (lines 30–38) interprets a string specification (e.g., `"default=4,embedding=2"`) and produces a mapping consumed by **`cq_mixed_params`** (lines 21–27). This allows the attention heads to remain at higher precision while feed-forward networks drop to 2-bit, balancing quality and size without retraining.

## Training-Aware vs. Deployment Quantization

### Quantization-Aware Training (QAT)

CQ seamlessly integrates into the fine-tuning loop via **`configure_qat`** (lines 56–58) and **`maybe_quant_weights`** (lines 87–92). When enabled, these functions simulate the forward-pass quantization error during LoRA training (`--qat-bits auto` as documented in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md)). The model learns to compensate for the rounding induced by the 2-bit codebook, ensuring that quantized weights retain task-specific accuracy after deployment.

### Runtime Deployment Quantization

For inference, **`configure_deploy`** accepts `act_bits` and `kv_bits` parameters to set the precision for activations and key-value caches, respectively. The **`deploy_quantize`** routine then bakes the quantized weights directly into the `.cact` format. When building via the CLI, the `--bits 2` flag triggers this pipeline, outputting a standalone binary that requires no external calibration data to run.

## Integration in the Needle Architecture

Quantization is injected transparently into the forward pass in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). The helper **`_aq`** (lines 22–25) wraps weight tensors with the active quantizer, while **`maybe_quant_kv`** (lines 42–44) handles KV-cache compression. Additionally, **`fake_quant_act`** and **`cq_fake_quant_kv`** (defined around lines 34–47 in [`quantize.py`](https://github.com/cactus-compute/needle/blob/main/quantize.py)) provide differentiable approximations used during QAT, ensuring that the same model code path executes for both training and inference without conditional branches.

## Key Benefits of Cactus Quants

- **Extreme size reduction**: Needle 2’s 45M parameters compress from hundreds of megabytes (FP16) to a **14MB binary** using CQ 2-bit weights.
- **Memory-efficient inference**: The entire model loads into approximately **28MB of RAM**, eliminating disk paging and enabling execution on microcontrollers.
- **Speed optimization**: Hadamard transforms operate in **O(n log n)** time via fast matrix multiplies, and 2-bit arithmetic reduces bandwidth pressure on hardware accelerators.
- **Accuracy preservation**: Sanity checks in [`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py) (lines 58–59) enforce that de-quantized weights maintain a **correlation >0.9** with their full-precision originals.
- **Flexible deployment**: Mixed-precision maps let developers assign higher bit-widths to sensitive layers (e.g., embeddings) while compressing bulk parameters aggressively.

## Code Examples

### Enable Deployment Quantization

```python
from needle.model import quantize as cq

# Configure 2-bit weights and 2-bit KV caches for minimal footprint

cq.configure_deploy(act_bits=2, kv_bits=2)
cq.deploy_quantize(model)

```

### Build a Quantized Model via CLI

```bash

# Bake LoRA adapters into a CQ2-bit binary

needle build checkpoints/needle2.pkl \
    --lora checkpoints/needle_lora.pkl \
    --out my_needle.cact \
    --bits 2

```

### Run Inference with Quantized Weights

```python
import needle

# The engine automatically handles dequantization; no extra flags needed

agent = needle.Needle(weights="my_needle.cact", tools=[...])
result = agent.run("What is the weather in Berlin?")

```

### Inspect Quantization Statistics

```python
from needle.model import quantize as cq

# Define a mixed-precision map: embeddings at 2-bit, rest at 4-bit

bits_map, default = cq.parse_bits_map("default=4,embedding=2")
stats = cq.cq_mixed_stats(params, bits_map, default)

print(f"Avg bits/weight: {stats[0]:.2f}, Size: {stats[1]:.2f} MiB")

```

## Key Source Files

| File | Purpose | Link |
|------|---------|------|
| [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) | Core CQ implementation: Hadamard rotation, codebooks, QAT hooks | [quantize.py](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) |
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | Injection points (`_aq`, `maybe_quant_kv`) for transparent quantization | [architecture.py](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) |
| [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) | Guide to quantization-aware training with `--qat-bits` | [finetuning.md](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) |
| [`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py) | Validation that de-quantized weights correlate >0.9 with originals | [test_build.py](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py) |
| [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) | High-level overview of CQ-compressed Needle 2 benchmarks | [README.md](https://github.com/cactus-compute/needle/blob/main/README.md) |

## Summary

- **Cactus Quants (CQ)** uses Hadamard-transformed, group-wise quantization with Lloyd-Max codebooks defined in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) to achieve 2-bit compression ratios.
- The pipeline supports both **quantization-aware training** (`configure_qat`, `maybe_quant_weights`) and **post-training deployment** (`configure_deploy`, `deploy_quantize`) workflows.
- Integration via `_aq` and `maybe_quant_kv` in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py) ensures the same model executes transparently with or without compression.
- CQ enables Needle 2 to deliver competitive accuracy in a 14MB binary that consumes only 28MB of RAM during inference, verified by correlation checks in [`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py).

## Frequently Asked Questions

### What bit widths does Cactus Quants support?

CQ supports configurable per-layer bit widths, typically **2-bit and 4-bit**, specified globally via `configure_deploy` or granularly through `parse_bits_map`. The underlying codebooks are pre-computed for these widths in `_cq_codebook_np`, allowing efficient selection at build time.

### How does CQ maintain accuracy at 2-bit precision?

By applying a **Hadamard rotation** before quantization to Gaussianize weight distributions, then mapping values to **Lloyd-Max centroids** optimized for the target bit-width, CQ minimizes reconstruction error. The test suite in [`tests/test_build.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_build.py) enforces that de-quantized weights retain a correlation greater than 0.9 with their full-precision counterparts.

### Can I fine-tune a model that is already CQ-quantized?

Yes. The **`configure_qat`** and **`maybe_quant_weights`** functions insert fake-quantization nodes during LoRA fine-tuning, allowing the model to adapt to the rounding error introduced by the 2-bit codebook. This quantization-aware training (QAT) workflow is detailed in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) using the `--qat-bits auto` flag.

### What is the memory footprint during inference?

A full session with Needle 2 runs in approximately **28MB of RAM**, including the 14MB binary containing 45M 2-bit parameters and additional working memory for activations. The CQ engine decompresses weights on-the-fly using fast Hadamard transforms without loading full-precision floats into memory.