# How Needle 2 Compresses to a 14MB Binary: The CQ Quantization Pipeline

> Discover how Needle 2 achieves a ~14MB binary size with CQ INT4 quantization, shared Lloyd-Max codebooks, and LSB-first index packing, enabling 4x reduction over FP16.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-09-08

---

**Needle 2 compresses its 45M-parameter model into a ~14MB binary using CQ (Cactus-Quant) INT4 quantization, a shared Lloyd-Max codebook, and LSB-first index packing, achieving a 4× size reduction over standard FP16 checkpoints.**

Needle 2 is a compact transformer implementation from the cactus-compute/needle repository designed for edge deployment. Unlike standard frameworks that store 45 million parameters in ~56MB of float16 weights, Needle 2 employs a custom compression stack that produces a self-contained `.cact` binary roughly one-fourth that size. The compression pipeline is implemented across [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) and [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), combining vector quantization with a lean file format to minimize both disk and memory footprints to approximately 14MB and 28MB respectively.


## The Three-Stage Compression Pipeline

The reduction to 14MB relies on three tightly-coupled stages: aggressive quantization of weight matrices, shared codebook generation, and bitwise packing of indices.

### Stage 1: CQ Quantization to INT4 and Mixed Precision

In [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), the `cq_quantize()` and `cq_quantize_params()` functions convert the bulk of matmul kernel weights from **float16 to CQ (Cactus-Quant) INT4**, with optional mixed-precision support for 2-bit or 3-bit groups. This quantization reduces raw weight storage by a factor of 4. Critical tensors—including **norms, Hadamard diagonals, and gate tensors**—remain in FP16 to preserve model accuracy. The `fake_quant()` helper simulates quantization during training to maintain convergence before the final export.

### Stage 2: Lloyd-Max Codebook Generation

Rather than storing full per-tensor scaling factors, Needle 2 builds a **Lloyd-Max codebook** per group (default group size of 128) using `_cq_codebook_np()` in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). This codebook is computed once during export and stored in the file header. Because the same codebook is reused for every quantized tensor, the format avoids the per-tensor overhead that plagues naive quantization schemes, reducing metadata to a few kilobytes regardless of model depth.

### Stage 3: LSB-First Packing and Layout

The final stage packs quantized indices into a contiguous byte stream using **LSB-first nibble packing** (e.g., packing two 4-bit indices per byte). In [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the `_pack_lsb()` and `_cq_pack()` routines (lines 84–100) serialize the indices alongside per-group FP16 norm vectors. The complementary `_cq_unpack()` routine (lines 102–118) allows the C++ runtime to reconstruct weights on-the-fly with a single fast kernel, keeping RAM usage low during inference.


## Inside the `.cact` File Format Structure

The `.cact` binary is a purpose-built format designed for zero-overhead loading and memory mapping. It consists of three distinct regions: a compact header, a tensor directory, and the raw tensor blobs.

### Header Metadata and Shared Codebook

The file begins with a **120-byte header** containing architecture metadata: vocabulary size, model dimensions, codebook length, KV-window configuration, and quantization parameters. The shared Lloyd-Max codebook is embedded immediately following this header. This design is implemented in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 12–18) and ensures that dequantization parameters are loaded once and reused for every tensor in the model.

### Tensor Directory Layout

Following the header is a **tensor directory** containing one fixed-size record per tensor. Each record stores dtype, shape, file offset, size, group identifier, and bit-width—without any tensor names. The tensor order is strictly predetermined (embedding → per-layer weights → engram tables → final norm → optional heads → tokenizer), eliminating the need for string storage and parsing overhead. See the directory layout definition in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 33–38).

### CQ Blob Storage

Weight tensors are stored as **CQ blobs**: packed integer indices followed by per-group FP16 norm vectors. Small non-quantized tensors (such as the final layer norm or specific bias vectors) are stored as plain FP16 or FP32 arrays. Because the codebook and norms are shared, the **total file size is dominated by the packed indices**, resulting in the ~14MB footprint for 45 million parameters.


## Exporting and Loading Compressed Models

The [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) module provides high-level APIs for creating and reading `.cact` files, while [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) handles runtime loading.

### Exporting a Checkpoint to `.cact`

To compress a trained checkpoint, use `write_export()` with the desired bit-width and group size:

```python
from needle.model.export import write_export
from needle.model.run import load_checkpoint
from needle.model.tokenizer import get_tokenizer

# Load training checkpoint

params, config, _ = load_checkpoint("ckpt/needle-45M.ckpt")

# Export with 4-bit CQ quantization (default)

write_export(
    params,
    config,
    path="needle.cact",        # ~14MB binary

    bits=4,                    # 4-bit CQ for weight tensors

    group=128,                 # Group size for codebook

    tokenizer=get_tokenizer(config.vocab_size),
)

```

The function builds the 120-byte header, serializes each tensor via `_cq_pack()`, and writes the contiguous binary stream.

### Loading at Runtime

The runtime loader reconstructs tensors on-demand, keeping memory usage at ~28MB:

```python
from needle.model.run import load_checkpoint
from needle.model.export import read_export

# Read .cact file

metadata, tensors = read_export("needle.cact")

# metadata contains architecture fields (num_layers, d_model, etc.)

# tensors is a list of NumPy arrays in the predetermined order

```

The `read_export()` function parses the header and directory, then uses `_cq_unpack()` to decompress each quantized weight only when needed.

### Inspecting Codebook Efficiency

You can verify the codebook sharing by inspecting the header:

```python
from needle.model.export import read_export

meta, _ = read_export("needle.cact")
print(f"Codebook entries: {meta['codebook'].shape[0]}")   # → 48 (2+3+4 bits)

```

Because the codebook is stored once in the header (as constructed in `_pack_cact()` at lines 47–53 of [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)), it occupies negligible space compared to the weight data.


## Summary

- **CQ (Cactus-Quant) INT4** reduces weight precision from FP16 to 4-bit integers, cutting storage by 75%.
- A **shared Lloyd-Max codebook** (computed in `_cq_codebook_np()`) is stored once in the 120-byte header and reused across all tensors.
- **LSB-first packing** (`_cq_pack()`) stores indices as packed nibbles/bytes with per-group FP16 norms, optimized for fast C++ decoding.
- The **`.cact` format** eliminates tensor names and uses fixed-order records, minimizing metadata overhead.
- The final 45M-parameter model occupies **~14MB on disk** and runs in **~28MB of RAM**, roughly 4× smaller than an FP16 equivalent.


## Frequently Asked Questions

### What is CQ (Cactus-Quant) quantization?

CQ quantization is a custom vector quantization scheme implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) that represents weight matrices as low-bit integers (INT4, INT3, or INT2) using a shared Lloyd-Max codebook. During inference, indices are mapped back to floating-point values by multiplying codebook centroids with per-group FP16 normalization vectors, allowing the model to retain accuracy while reducing weight storage by a factor of 4 or more.

### Why is the `.cact` file format limited to 14MB for 45M parameters?

The 14MB footprint results from the combination of 4-bit quantization (reducing the 56MB FP16 baseline to ~14MB) and an extremely compact file structure. By storing a single shared codebook in the header, omitting tensor name strings, and using LSB-first bit packing for indices, the format ensures that nearly every byte of the 14MB file stores actual parameter data rather than metadata.

### How does Needle 2 keep RAM usage at ~28MB when the model is 45M parameters?

Unlike frameworks that decompress the entire model into FP16 at load time, Needle 2 keeps weights in their compressed CQ format and dequantizes them on-the-fly during the forward pass. The `_cq_unpack()` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) reconstructs weight blocks lazily using the shared codebook, ensuring that only the current layer’s activations and a small buffer of dequantized weights reside in memory simultaneously.

### Can I adjust the compression level or bit-width?

Yes. The `write_export()` function accepts a `bits` parameter (default 4) and a `group` parameter (default 128) to control the quantization granularity. You can specify mixed-precision configurations (combining 2-bit, 3-bit, and 4-bit groups) by modifying the quantization configuration before export, though the default 4-bit setting provides the optimal balance for the 45M-parameter model to stay within the 14MB target.