What Is Cactus Quants (CQ) Quantization and Its Benefits

Cactus Quants (CQ) is a bespoke quantization pipeline that compresses neural network weights to as low as 2-bit precision using Hadamard transforms, group-wise Lloyd-Max codebooks, and training-aware fine-tuning, enabling the 45M-parameter Needle 2 model to ship as a 14MB binary and run entirely in 28MB of RAM.

Cactus Quants (CQ) is the proprietary compression engine developed by cactus-compute for the Needle repository. It powers the extreme efficiency of Needle 2 by applying advanced linear algebra and learned quantization tables to reduce model size by up to 25× compared to full-precision floats. The following sections break down the architecture of CQ, its implementation across needle/model/quantize.py and needle/model/architecture.py, and the concrete performance gains it delivers on edge devices.

Core Mechanics of CQ Quantization

Group-Wise Quantization with Hadamard Rotation

At the heart of CQ is group-wise quantization that processes weights in fixed-size blocks (default group_size of 128) rather than tensor-wide scales. Before quantization, each block is rotated by a pre-computed Walsh-Hadamard matrix (_cq_hadamard_np). This transform spreads information uniformly, Gaussianizing the distribution and minimizing quantization error for low-bit representations. The rotation and scaling logic resides in cq_quantize (approximately lines 34–48 in needle/model/quantize.py), while the inverse operation reconstructs weights on-the-fly during inference.

Learned Lloyd-Max Codebooks

CQ does not use uniform linear quantization. Instead, it generates Lloyd-Max derived codebooks (_cq_codebook_np, lines 13–16) that contain optimal centroids for each target bit-width (e.g., 2-bit or 4-bit). These codebooks are computed once per bit-width and shared across all compatible layers, ensuring that the most frequent weight values retain the highest fidelity while outliers are mapped to the nearest centroid.

Mixed-Precision Bit Mapping

To preserve accuracy in critical layers while aggressively compressing others, CQ supports per-layer bit-width configuration. The function parse_bits_map (lines 30–38) interprets a string specification (e.g., "default=4,embedding=2") and produces a mapping consumed by cq_mixed_params (lines 21–27). This allows the attention heads to remain at higher precision while feed-forward networks drop to 2-bit, balancing quality and size without retraining.

Training-Aware vs. Deployment Quantization

Quantization-Aware Training (QAT)

CQ seamlessly integrates into the fine-tuning loop via configure_qat (lines 56–58) and maybe_quant_weights (lines 87–92). When enabled, these functions simulate the forward-pass quantization error during LoRA training (--qat-bits auto as documented in doc/finetuning.md). The model learns to compensate for the rounding induced by the 2-bit codebook, ensuring that quantized weights retain task-specific accuracy after deployment.

Runtime Deployment Quantization

For inference, configure_deploy accepts act_bits and kv_bits parameters to set the precision for activations and key-value caches, respectively. The deploy_quantize routine then bakes the quantized weights directly into the .cact format. When building via the CLI, the --bits 2 flag triggers this pipeline, outputting a standalone binary that requires no external calibration data to run.

Integration in the Needle Architecture

Quantization is injected transparently into the forward pass in needle/model/architecture.py. The helper _aq (lines 22–25) wraps weight tensors with the active quantizer, while maybe_quant_kv (lines 42–44) handles KV-cache compression. Additionally, fake_quant_act and cq_fake_quant_kv (defined around lines 34–47 in quantize.py) provide differentiable approximations used during QAT, ensuring that the same model code path executes for both training and inference without conditional branches.

Key Benefits of Cactus Quants

• Extreme size reduction: Needle 2’s 45M parameters compress from hundreds of megabytes (FP16) to a 14MB binary using CQ 2-bit weights.
• Memory-efficient inference: The entire model loads into approximately 28MB of RAM, eliminating disk paging and enabling execution on microcontrollers.
• Speed optimization: Hadamard transforms operate in O(n log n) time via fast matrix multiplies, and 2-bit arithmetic reduces bandwidth pressure on hardware accelerators.
• Accuracy preservation: Sanity checks in tests/test_build.py (lines 58–59) enforce that de-quantized weights maintain a correlation >0.9 with their full-precision originals.
• Flexible deployment: Mixed-precision maps let developers assign higher bit-widths to sensitive layers (e.g., embeddings) while compressing bulk parameters aggressively.

Code Examples

Enable Deployment Quantization

from needle.model import quantize as cq

# Configure 2-bit weights and 2-bit KV caches for minimal footprint

cq.configure_deploy(act_bits=2, kv_bits=2)
cq.deploy_quantize(model)

Build a Quantized Model via CLI

# Bake LoRA adapters into a CQ2-bit binary

needle build checkpoints/needle2.pkl \
    --lora checkpoints/needle_lora.pkl \
    --out my_needle.cact \
    --bits 2

Run Inference with Quantized Weights

import needle

# The engine automatically handles dequantization; no extra flags needed

agent = needle.Needle(weights="my_needle.cact", tools=[...])
result = agent.run("What is the weather in Berlin?")

Inspect Quantization Statistics

from needle.model import quantize as cq

# Define a mixed-precision map: embeddings at 2-bit, rest at 4-bit

bits_map, default = cq.parse_bits_map("default=4,embedding=2")
stats = cq.cq_mixed_stats(params, bits_map, default)

print(f"Avg bits/weight: {stats[0]:.2f}, Size: {stats[1]:.2f} MiB")

Key Source Files

File	Purpose	Link
needle/model/quantize.py	Core CQ implementation: Hadamard rotation, codebooks, QAT hooks	quantize.py
needle/model/architecture.py	Injection points (_aq, maybe_quant_kv) for transparent quantization	architecture.py
doc/finetuning.md	Guide to quantization-aware training with --qat-bits	finetuning.md
tests/test_build.py	Validation that de-quantized weights correlate >0.9 with originals	test_build.py
README.md	High-level overview of CQ-compressed Needle 2 benchmarks	README.md

Summary

• Cactus Quants (CQ) uses Hadamard-transformed, group-wise quantization with Lloyd-Max codebooks defined in needle/model/quantize.py to achieve 2-bit compression ratios.
• The pipeline supports both quantization-aware training (configure_qat, maybe_quant_weights) and post-training deployment (configure_deploy, deploy_quantize) workflows.
• Integration via _aq and maybe_quant_kv in architecture.py ensures the same model executes transparently with or without compression.
• CQ enables Needle 2 to deliver competitive accuracy in a 14MB binary that consumes only 28MB of RAM during inference, verified by correlation checks in tests/test_build.py.

Frequently Asked Questions

What bit widths does Cactus Quants support?

CQ supports configurable per-layer bit widths, typically 2-bit and 4-bit, specified globally via configure_deploy or granularly through parse_bits_map. The underlying codebooks are pre-computed for these widths in _cq_codebook_np, allowing efficient selection at build time.

How does CQ maintain accuracy at 2-bit precision?

By applying a Hadamard rotation before quantization to Gaussianize weight distributions, then mapping values to Lloyd-Max centroids optimized for the target bit-width, CQ minimizes reconstruction error. The test suite in tests/test_build.py enforces that de-quantized weights retain a correlation greater than 0.9 with their full-precision counterparts.

Can I fine-tune a model that is already CQ-quantized?

Yes. The configure_qat and maybe_quant_weights functions insert fake-quantization nodes during LoRA fine-tuning, allowing the model to adapt to the rounding error introduced by the 2-bit codebook. This quantization-aware training (QAT) workflow is detailed in doc/finetuning.md using the --qat-bits auto flag.

What is the memory footprint during inference?

A full session with Needle 2 runs in approximately 28MB of RAM, including the 14MB binary containing 45M 2-bit parameters and additional working memory for activations. The CQ engine decompresses weights on-the-fly using fast Hadamard transforms without loading full-precision floats into memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}