# Benefits of CQ2-Bit Quantization in Needle 2: Extreme Compression for Edge AI

> Discover the benefits of CQ2-bit quantization in Needle 2 for extreme model compression. Achieve 8-10x size reduction for Edge AI while maintaining accuracy.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-28

---

**Needle 2 leverages CQ 2-bit (Cactus-Quants) quantization to compress a 45-million-parameter model into a 14 MB binary that runs in just 28 MB of RAM, delivering 8–10× size reduction over fp16 while maintaining accuracy through Lloyd-Max optimized codebooks and Walsh-Hadamard transforms.**

The **cactus-compute/needle** repository implements a novel quantization scheme that stores and executes transformer weights in a ternary 2-bit format. Unlike conventional 8-bit or 16-bit quantization, CQ 2-bit quantization enables tiny binary sizes, memory-mapped inference, and fast O(n log n) arithmetic without learned parameters. This architecture makes Needle 2 ideal for air-gapped environments, edge devices, and any deployment scenario where storage and RAM are constrained.

## Extreme Model Compression with Ternary Codebooks

Needle 2’s export pipeline packs the entire model into a single **14 MB** `.cact` file by representing weight tensors as CQ 2-bit integers. This achieves roughly **4× compression** compared to 8-bit formats and **8–10× reduction** versus standard fp16 storage.

In [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the export routine (lines 62–68) groups weights and applies a **2-bit codebook** with per-group L2 normalization. Each weight group is encoded using ternary values that reference a shared codebook, eliminating the need to store full-precision floats. The `cact` binary header (lines 19–30) encodes the bit-width metadata (`bits` and `kv_bits` fields), allowing the runtime to interpret the quantized buffer correctly without recompilation.

## Minimal Runtime Memory via Memory-Mapping

A full inference session in Needle 2 consumes approximately **28 MB** of RAM, achieved through aggressive quantization and zero-copy weight loading.

The runtime loads the `.cact` blob as a **read-only memory-mapped file**, as implemented in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 12–17). Because the quantized weights remain in their compressed 2-bit representation throughout inference, the engine avoids the memory duplication typically required for on-the-fly dequantization. Only a small KV cache is allocated in writable memory, keeping the working set minimal.

## Fast Arithmetic Using Walsh-Hadamard Transforms

CQ 2-bit quantization pairs with a fixed **Walsh-Hadamard** matrix to accelerate matrix-vector products to **O(n log n)** complexity.

In [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 41–45), the `cq_quantize` routine applies the orthogonal transform in-place on each weight group (`rot = groups @ H`) before normalization. During inference, this structure allows the compute kernel to use cheap lookup operations against the 2-bit codebook rather than expensive floating-point multiplications. The transform requires no learned parameters and is applied uniformly across all groups, ensuring consistent latency characteristics.

## Accuracy Preservation Through Lloyd-Max Optimization

Despite aggressive compression to 2 bits, Needle 2 maintains competitive accuracy by optimizing the quantization codebook for minimal distortion.

The `_cq_codebook_np` utility in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 31–40) generates a **Lloyd-Max optimized** codebook designed for unit-sphere weight distributions. By minimizing the expected quantization error (measured via `noise_scale` and `cq_distortion` utilities), each weight is mapped to its nearest centroid with low relative error. This codebook is computed once per bit-width and reused across all groups, ensuring that the ternary representation preserves the model's semantic capabilities.

## Cross-Platform Deployment and KV-Cache Control

The CQ 2-bit format enables uniform deployment across CPU, GPU, and Metal backends without platform-specific recompilation.

The `.cact` file header encodes both weight precision (`bits`) and KV-cache precision (`kv_bits`), allowing independent configuration of storage and activation quantization. As defined in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 61–66), the `configure_deploy` function supports setting `kv_bits` separately from the main weights. When `kv_bits` is set below 8, the KV cache also uses CQ 2-bit compression, further reducing memory for long-context inference while keeping weights at 2-bit precision.

## How to Export and Run CQ 2-Bit Models

You can generate a CQ 2-bit model using the Needle CLI or Python API. The following examples demonstrate loading a pre-quantized model and explicitly configuring 2-bit export:

```python

# Load a CQ 2-bit model (default exported format)

import needle
agent = needle.Needle(weights="needle2.cact", tools=[...])
response = agent.run("What is the weather in Nairobi?")
print(response["results"])

```

To explicitly force 2-bit quantization during export, use the `--bits` flag which triggers `configure_deploy` in the quantization pipeline:

```bash

# Export with CQ 2-bit quantization

needle build base.pkl --out needle2.cact --bits 2

```

The export script handles flag parsing and invokes `configure_deploy` to bake the 2-bit codebook into the final binary, ensuring the runtime reads weights directly without conversion.

## Summary

- **CQ 2-bit quantization** reduces a 45M-parameter Needle 2 model to 14 MB on disk (8–10× vs fp16).
- **Memory-mapped loading** keeps runtime RAM at approximately 28 MB by avoiding weight decompression copies.
- **Walsh-Hadamard transforms** enable O(n log n) inference speed without learned parameters.
- **Lloyd-Max codebooks** minimize quantization distortion to preserve model accuracy.
- **Independent KV-cache precision** (`kv_bits`) allows further memory savings for long sequences.
- **Uniform binary format** runs on any platform (CPU, GPU, Metal) without recompilation.

## Frequently Asked Questions

### What does CQ stand for in Needle 2’s quantization scheme?

**CQ stands for Cactus-Quants**, the custom quantization format developed for the cactus-compute/needle repository. It refers specifically to the ternary 2-bit codebook compression used to store weight tensors and KV caches in a compact, fixed-point representation.

### How does CQ 2-bit quantization affect model accuracy compared to fp16?

Needle 2 maintains competitive accuracy through **Lloyd-Max optimization** of the quantization codebook. As implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), the codebook is pre-computed to minimize distortion for unit-sphere weight distributions, ensuring each 2-bit centroid represents the nearest full-precision value with low relative error.

### Can CQ 2-bit models run on hardware other than CPU?

Yes. The `.cact` binary format is **platform-agnostic**; the same file runs on CPU, GPU, or Metal backends without recompilation. The header specifies the bit-width (`bits` and `kv_bits`), allowing the C++ inference engine to select the appropriate numeric path automatically.

### Is it possible to use different bit-widths for weights and the KV cache?

Yes. Needle 2 supports **decoupled precision control** via the `kv_bits` parameter in `configure_deploy` ([`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), lines 61–66). You can keep weights at 2-bit while configuring the KV cache to higher precision (or vice versa), trading off between memory usage and context-window stability.