# CQ2-Bit Weight Compression in Needle 2: How Cactus Quants Achieves 14MB Model Size

> Discover CQ2-bit weight compression in Needle 2, a custom technique using Cactus Quants to achieve a 14MB model size with 2 bits per weight. Learn how it works.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-12

---

**CQ2-bit weight compression is a custom quantization technique called Cactus Quants (CQ) that packs Needle 2's weight matrices into 2 bits per weight using Hadamard rotation and Lloyd-Max codebook optimization, enabling the 45M-parameter model to fit in a single 14MB binary.**

Needle 2, the inference engine from the `cactus-compute/needle` repository, achieves its remarkably small footprint through aggressive weight compression. The system stores its entire 45-million-parameter model in just 14 megabytes by employing a proprietary quantization scheme known as **Cactus Quants (CQ)**. This technique reduces storage to roughly one-quarter of an FP16 baseline while maintaining inference quality.

## What Is CQ2-Bit Weight Compression?

**CQ2-bit** refers to the specific configuration of Cactus Quants that quantizes model weights to **2 bits per weight**. Unlike standard 8-bit or 16-bit floating-point formats, this custom codec processes weights in fixed-size groups and applies mathematical rotation to maximize quantization efficiency. The compression is lossy but optimized to preserve the model's ability to perform tool-calling and structured extraction tasks.

The implementation centers on a group size of 128 values, defined by the constant `CQ_GROUP_SIZE = 128` in the source. For each group, the algorithm applies a Hadamard transform, maps the rotated values to a compact codebook, then reverses the rotation to produce the final compressed representation.

## How the CQ2-Bit Compression Pipeline Works

The compression logic resides in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) and follows a five-stage pipeline:

### Grouping and Padding

The algorithm first ensures tensor dimensions align with the group size. The code calculates padding using `pad = (-D) % g` where `D` is the dimension and `g` is the group size (128). If padding is required, the system applies `jnp.pad(w, ...)` to extend the tensor, creating uniform groups for processing.

### Hadamard Rotation

Each group undergoes rotation via a normalized Hadamard matrix generated by `_cq_hadamard_np`. The operation `rot = groups @ H` transforms the weight distribution to reduce dynamic range before quantization. This rotation step is critical for concentrating values around zero, which improves the accuracy of low-bit encoding.

### Codebook Selection and Quantization

The system selects a codebook using `_cq_codebook_np(bits, group_size)`, where `bits = 2` for CQ2-bit mode. For 2-bit quantization, the codebook is generated either through ternary optimization or Lloyd-Max Gaussian clustering (`_lloyd_max_gaussian`). The quantization step uses nearest-neighbor mapping: vectors are normalized with `unit = rot / jnp.maximum(norm, 1e-12)`, then snapped to the closest codebook entry via `_cq_nearest(unit, cb)`.

### De-rotation and Trimming

After quantization, the values pass through the inverse Hadamard transform `( ... ) @ H` to restore original orientation. The system then removes any padding added during the grouping phase using `return deq[..., :D] if pad else deq`, yielding the final compressed weights.

## Implementation in the Needle 2 Codebase

The CQ2-bit system is orchestrated through specific APIs and configuration files:

| File | Role |
|------|------|
| [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) | Core implementation containing `cq_quantize`, `cq_quantize_params`, and rotation matrices |
| [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) | Handles model export with `weight_bits` specification and constructs the "CQ W2" label |
| [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) | Runtime loader (`needle_load`) that streams and decompresses weights on-demand |

To enable CQ2-bit during deployment, the code constructs a configuration object with `weight_bits="2"` and passes it to `deploy_quantize`. The function signature accepts parameters alongside this config: `deploy_quantize(params, config=...)`. The default quantization settings are established via `configure_deploy(act_bits=8, kv_bits=8, kv_group=64)`, which prepares the quantization context before the 2-bit weight conversion occurs.

## Enabling CQ2-Bit Compression in Practice

To load a pre-compressed Needle 2 model using the CQ2-bit binary format:

```python
import needle

# Load a .cact file exported with weight_bits="2"

agent = needle.Needle(
    tools="[]",
    weights="acme/needle-2.cact"  # CQ2-bit compressed binary

)

result = agent("Summarize the last news headlines.")
print(result)

```

To manuallyquantize a model using the CQ2-bit pipeline during export:

```python
from needle.model.quantize import configure_deploy, deploy_quantize

# Configure activation and KV cache quantization

configure_deploy(act_bits=8, kv_bits=8, kv_group=64)

# Apply CQ2-bit weight compression

quantized_params, label = deploy_quantize(
    params,
    config=type("Cfg", (), {"weight_bits": "2"})()  # Trigger CQ2-bit

)

print(label)  # Output: "CQ W2"

# Save quantized_params to a .cact file for distribution

```

## Memory and Performance Benefits

The CQ2-bit format delivers substantial efficiency gains:

- **Storage Footprint**: Reduces the 45M-parameter model to 14MB, compared to approximately 90MB for FP16 weights (a ~6x reduction).
- **Runtime RAM**: The engine requires only ~28MB of RAM during inference because weights are streamed and decompressed on-the-fly via `needle_load` in [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py).
- **Latency**: Hadamard transforms and small codebook lookups operate efficiently on modest hardware, with dequantization happening lazily during token generation.

## Summary

- **CQ2-bit** stands for Cactus Quants 2-bit, a custom quantization method using 2 bits per weight.
- The algorithm processes weights in **groups of 128**, applying Hadamard rotation and Lloyd-Max codebook optimization.
- Implementation lives in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), with APIs exposed through `configure_deploy` and `deploy_quantize`.
- **45 million parameters** compress into **14MB**, requiring only **28MB RAM** at runtime.
- Compression is triggered by setting `weight_bits="2"` in the deployment configuration.

## Frequently Asked Questions

### What does CQ stand for in Needle 2's compression system?

**CQ** stands for **Cactus Quants**, the proprietary quantization framework developed for the Needle inference engine. It encompasses multiple bit-width configurations, with CQ2-bit specifically referring to the 2-bit-per-weight variant used in Needle 2's smallest deployment targets.

### How does CQ2-bit compression achieve 2 bits per weight without significant accuracy loss?

The system uses **Hadamard rotation** to redistribute weight values before quantization, making them more amenable to low-bit encoding. After rotation, a Lloyd-Max optimized codebook (or ternary codebook) represents the values using only four possible states (2 bits). The de-rotation step restores the original vector orientations, minimizing reconstruction error during inference.

### What is the file size reduction compared to standard FP16 weights?

Needle 2's 45-million-parameter model occupies **14MB** with CQ2-bit compression, compared to approximately **90MB** in FP16 format. This represents an 84% reduction in storage size, or roughly one-sixth of the original footprint, while maintaining functional capability for tool-calling and structured generation tasks.

### Which API functions control CQ2-bit compression during model export?

The compression pipeline is controlled by two primary functions from `needle.model.quantize`: `configure_deploy()` sets the global quantization context for activations and KV caches, while `deploy_quantize()` applies the specific weight compression when passed a configuration object with `weight_bits="2"`. The resulting quantized parameters are then serialized through the export pipeline in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).