# How CQ2-bit Quantization Contributes to Needle 2's Small Footprint

> Discover how CQ2-bit quantization shrinks Needle 2 to a 14 MiB binary, running 45M parameters in just 28 MiB RAM. Optimize your model size efficiently.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-06

---

**CQ2-bit quantization enables Needle 2 to compress a 45-million-parameter model into a single 14 MiB binary that runs full sessions in approximately 28 MiB of RAM.**

Needle 2, developed by Cactus Compute, achieves its remarkable efficiency through a custom blockwise quantization scheme called Cactus-Quants (CQ). This article examines the technical mechanisms behind CQ2-bit quantization and how they deliver a 6× size reduction compared to full-precision storage.

## What is CQ2-bit Quantization?

CQ2-bit quantization is a **group-wise weight compression technique** that stores neural network parameters using only 2 bits per weight instead of the standard 16-bit floating-point format. The implementation lives in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), which defines the core `cq_quantize` function and supporting infrastructure.

The quantizer supports multiple bit widths through the `CB_BITS = (2, 3, 4)` constant, but Needle 2 specifically targets **2-bit compression** for maximum deployment efficiency.

## Blockwise Quantization Architecture

The CQ quantizer operates on **groups of 128 weights**, applying a sequence of deterministic transformations that preserve model quality while minimizing storage.

### Hadamard Rotation

Each weight group undergoes rotation by a fixed **Hadamard matrix** (`_cq_hadamard_np`), an orthonormal transform implemented in lines 119-124 of [`quantize.py`](https://github.com/cactus-compute/needle/blob/main/quantize.py):

```python
def cq_quantize(w, bits, group_size=128, codebook=None):
    cb = codebook if codebook is not None else jnp.asarray(_cq_codebook_np(bits, group_size))
    # Pad to multiple of group_size → reshape → Hadamard transform → quantize → inverse transform

    # ...

    return deq[..., :D] if pad else deq

```

The Hadamard matrix is **analytically defined and parameter-free**. This means zero storage overhead for the transform itself—a critical advantage over learned quantization methods.

### Static Codebook Quantization

After rotation, each component is quantized to the nearest entry in a **precomputed codebook**. For 2-bit quantization, the codebook contains exactly 4 entries.

The 2-bit codebook is generated by **Lloyd-Max clustering of a Gaussian distribution** (`_lloyd_max_gaussian` in lines 95-106). This deterministic construction means:

- The same codebook is reused across all layers
- No per-layer codebook storage is required
- Inference remains **bit-exact across platforms**

## Per-Group Storage Breakdown

CQ2-bit quantization achieves compression through a careful separation of structural and scale information. For each group of 128 weights:

- **2 bits per weight** for the codebook index (the structural information)
- **16-bit float for the per-group L2 norm** (the scale information)
- **No storage for the Hadamard matrix** (implicit, fixed transform)

This produces the exact formula implemented in `cq_model_bytes` (lines 75-88):

```python

# Estimate size after quantization

from needle.model.quantize import cq_model_bytes

size_bytes = cq_model_bytes(quantized_params, bits=2)

```

For a 45-million-parameter model, this yields approximately **14 MiB**—matching the size advertised in the README—versus ~90 MiB for FP16 storage.

## Integration with the Export Pipeline

The quantizer connects to deployment through two key integration points in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) and [`quantize.py`](https://github.com/cactus-compute/needle/blob/main/quantize.py).

### Configuration and Quantization

The `configure_deploy` function sets `_WEIGHT_BITS = 2` (lines 56-63), which `quantize_params_configured` uses during export:

```python
from needle.model.quantize import configure_deploy, quantize_params_configured

# Configure for 2-bit weight quantization

configure_deploy(act_bits=8, kv_bits=0, kv_group=64)
params_q2 = quantize_params_configured(original_params)

```

### Binary Serialization

The `export()` function in [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py) (lines 51-58) writes the final `.cact` blob containing:

1. Packed 2-bit indices (consecutive weights packed into bytes)
2. Per-group FP16 norms (appended after the index data)

```python
from needle.model.export import export

export(
    params=params_q2,
    config=my_config,
    out_path="needle2_2bit.cact",
    bits="2"
)

```

At runtime, [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) loads this compact blob, with the small size directly reducing RAM pressure during inference.

## Architectural Benefits of CQ2-bit Design

Beyond raw compression, CQ2-bit quantization provides structural advantages that align with Needle 2's design goals.

**Zero parameter overhead.** The Hadamard transform requires no learned or stored parameters, avoiding the "compression tax" common in other quantization schemes.

**Fast O(N log N) operations.** Quantization and dequantization reduce to matrix multiplications with the Hadamard matrix, which can be implemented efficiently using Fast Walsh-Hadamard Transform techniques.

**Deterministic cross-platform inference.** Static codebooks eliminate non-determinism from learned quantization tables, ensuring identical outputs across deployment targets.

**Efficient memory bandwidth.** The 2-bit representation reduces weight memory traffic by 8× during inference, critical for edge devices with constrained memory subsystems.

## Summary

- CQ2-bit quantization compresses Needle 2's 45M parameters into 14 MiB by storing **2 bits per weight** plus **minimal per-group metadata**
- The [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) implementation uses **Hadamard rotation** to spread energy uniformly before **static codebook quantization**
- Per-group storage includes 2-bit indices and a 16-bit FP16 norm—no other overhead
- The `cq_model_bytes` function and export pipeline in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) ensure accurate size prediction and efficient serialization
- These mechanisms combine to deliver **6× compression** with preserved model capability, enabling Needle 2's tiny deployment footprint

## Frequently Asked Questions

### What does CQ stand for in CQ2-bit quantization?

CQ stands for **Cactus-Quants**, the custom quantization framework developed by Cactus Compute specifically for the Needle model family. It is not a general industry standard like INT8 or FP4, but rather an optimized blockwise scheme designed for small language model deployment.

### Why 2 bits specifically rather than 4 or 8 bits?

Needle 2 targets **extreme edge deployment** where binary size and RAM footprint are the primary constraints. While CQ supports 2, 3, and 4 bits (via `CB_BITS`), the 2-bit mode achieves the critical 14 MiB threshold that enables single-file distribution. The Hadamard rotation makes 2-bit quantization viable by ensuring energy is distributed evenly across components, minimizing quantization error.

### How does CQ2-bit quantization affect inference speed?

Dequantization overhead is minimal due to the **Fast Walsh-Hadamard Transform** structure. The `rot @ H` operation runs in O(N log N) time, and the 8× reduction in memory bandwidth often yields net speedup on memory-bound inference workloads. The fixed codebook also eliminates lookup table cache misses compared to learned codebook approaches.

### Can CQ2-bit quantization be applied to other models?

The CQ quantizer in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) is implemented generally for JAX parameter dictionaries and could theoretically apply to other architectures. However, the 2-bit mode is specifically tuned and validated for Needle 2's "Simple Attention Network" architecture. Applying it to other models would require recalibration of the quantization-aware training or post-training quantization pipeline.