# CQ2-Bit Quantization in Needle 2: Compressing 45M Parameters into 14MB

> Discover CQ2-bit quantization in Needle 2, compressing 45M parameters to 14MB. Learn how 2-bit codebooks and Hadamard transforms enable efficient inference on low-RAM devices.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-21

---

**CQ2-bit quantization is the Cactus Quants (CQ) group-wise compression scheme that reduces Needle 2's 45 million parameters to a 14MB binary using 2-bit codebooks, Hadamard transforms, and stochastic rounding, enabling inference on devices with approximately 28MB of RAM.**

Needle 2 from the `cactus-compute/needle` repository achieves extreme model compression through **CQ2-bit quantization**, a technique that quantizes each weight to just two bits while preserving the performance of the full-precision model. By implementing **group-wise processing** with orthogonal rotations and learned codebooks in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), this method produces compact `.cact` binaries that require only 2 bits per weight plus minimal overhead. The result is a deployable model that maintains tool-calling capabilities while operating within severe memory constraints typical of microcontrollers and edge devices.

## How CQ2-Bit Quantization Works

The CQ2-bit scheme combines four technical components to minimize distortion when compressing 32-bit floats into 2-bit representations.

### Group-Wise Quantization

The algorithm splits each weight tensor into fixed-size **groups** (default **group size** of 128) and quantizes each group independently. Processing smaller groups keeps quantization error localized and manageable, allowing coarse 2-bit representations to approximate high-precision values with minimal accuracy loss compared to full-tensor quantization.

### Hadamard Transform

Before quantization, each group undergoes a **Walsh-Hadamard transform** using an orthogonal matrix `H`. This rotation spreads weight information uniformly across the vector, reducing the dynamic range and ensuring that a simple 4-level **2-bit codebook** can effectively represent the transformed values. The **Hadamard transform** is critical for making ultra-low-bit quantization feasible without significant performance degradation.

### Learned Codebooks

Quantization maps transformed weights to discrete values using static lookup tables built offline. The **codebooks** are constructed via the **Lloyd-Max algorithm** (k-means) on Gaussian-distributed data, generating four centroids for the 2-bit case. During inference, each weight is replaced by its nearest centroid index (2 bits), then rescaled by the group's norm to reconstruct the approximate value.

### Stochastic Rounding and Straight-Through Estimator

Training stability relies on the **`cq_ste`** (**straight-through estimator**) and **`add_cq_noise`** functions. During forward passes, the **stochastic rounding** process applies the quantizer but allows gradients to flow through the original high-precision weights. Additionally, calibrated Gaussian **noise injection** before quantization improves the model's tolerance to the aggressive compression, ensuring the learned weights remain robust when deployed at 2-bit precision.

## Needle 2 Implementation Pipeline

The export pipeline in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) orchestrates the conversion from full-precision checkpoints to compressed binaries through three stages.

### Configuration

Users specify target precision via `configure_deploy()` in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). Setting `act_bits=2`, `kv_bits=2`, and `kv_group=64` prepares the system for 2-bit activation and key-value cache quantization:

```python
from needle.model.quantize import configure_deploy

configure_deploy(act_bits=2, kv_bits=2, kv_group=64)

```

### Parameter Quantization

The **`cq_quantize_params()`** function iterates over all weight tensors, applying the Hadamard rotation, looking up nearest codebook entries, and rescaling by group norms. This function accepts `bits=2` and `group_size=128` parameters to control the compression level:

```python
from needle.model.quantize import cq_quantize_params

quantized_params = cq_quantize_params(
    base.model.params, 
    bits=2, 
    group_size=128
)

```

### Binary Export

The `export_cact()` function writes the quantized parameters to disk, while the CLI command `needle build --bits 2` triggers the full export sequence. According to the [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md), this produces a `.cact` binary containing all quantized weights and scaling factors required by the inference engine.

## Practical Usage Example

The following workflow demonstrates loading a base checkpoint, configuring 2-bit quantization, and exporting the compressed model:

```python
import needle
from needle.model.quantize import configure_deploy, cq_quantize_params
from needle.model.export import export_cact

# Load base checkpoint

base = needle.Needle(weights="cactus-needle/needle2")

# Configure for 2-bit deployment

configure_deploy(act_bits=2, kv_bits=2, kv_group=64)

# Quantize parameters (typically invoked via CLI)

quantized_params = cq_quantize_params(
    base.model.params, 
    bits=2, 
    group_size=128
)

# Export to .cact binary

export_cact(quantized_params, out_path="my_needle_2bit.cact")

```

## Summary

- **CQ2-bit quantization** compresses Needle 2's 45M parameters into a 14MB binary suitable for ~28MB RAM environments.
- The technique uses **group-wise processing** (groups of 128) with **Hadamard transforms** to prepare weights for aggressive compression.
- **Learned codebooks** generated via the **Lloyd-Max algorithm** map values to 2-bit indices (4 levels) with minimal distortion.
- **`cq_ste`** and **noise injection** enable training stability despite ultra-low precision.
- The implementation resides primarily in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), with export logic in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) and API documentation in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md).

## Frequently Asked Questions

### How much memory does CQ2-bit quantization save?

CQ2-bit quantization reduces the model from its full-precision size to a 14MB binary that requires approximately 28MB of RAM during inference. This represents roughly a 10x or greater compression ratio compared to standard 32-bit floating-point representations, enabling deployment on microcontrollers and other memory-constrained devices.

### What is the Hadamard transform's role in CQ2-bit quantization?

The **Hadamard transform** rotates weight groups using an orthogonal matrix before quantization, distributing information uniformly across the vector components. This preprocessing step reduces the effective dynamic range of the data, allowing a simple 4-entry **2-bit codebook** to accurately approximate the original values without significant loss of model capability.

### How does Needle 2 maintain accuracy with only 2 bits per weight?

Accuracy is preserved through **group-wise quantization** (keeping groups small at 128 elements), **learned Lloyd-Max codebooks** optimized for the weight distribution, and **noise injection** during training that improves robustness to quantization error. The **straight-through estimator** (`cq_ste`) further ensures gradients flow properly during backpropagation, allowing the network to learn weights that perform well when compressed.

### What file format stores the quantized model?

Needle 2 saves CQ2-bit quantized models as **`.cact` files**, a custom binary format generated by `export_cact()` in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py). These files contain the quantized weight indices, group scaling factors, and codebook references necessary for the inference engine to reconstruct activations during model execution.