How Needle 2 Compresses to a 14MB Binary: The CQ Quantization Pipeline

Needle 2 compresses its 45M-parameter model into a ~14MB binary using CQ (Cactus-Quant) INT4 quantization, a shared Lloyd-Max codebook, and LSB-first index packing, achieving a 4× size reduction over standard FP16 checkpoints.

Needle 2 is a compact transformer implementation from the cactus-compute/needle repository designed for edge deployment. Unlike standard frameworks that store 45 million parameters in ~56MB of float16 weights, Needle 2 employs a custom compression stack that produces a self-contained .cact binary roughly one-fourth that size. The compression pipeline is implemented across needle/model/quantize.py and needle/model/export.py, combining vector quantization with a lean file format to minimize both disk and memory footprints to approximately 14MB and 28MB respectively.

The Three-Stage Compression Pipeline

The reduction to 14MB relies on three tightly-coupled stages: aggressive quantization of weight matrices, shared codebook generation, and bitwise packing of indices.

Stage 1: CQ Quantization to INT4 and Mixed Precision

In needle/model/quantize.py, the cq_quantize() and cq_quantize_params() functions convert the bulk of matmul kernel weights from float16 to CQ (Cactus-Quant) INT4, with optional mixed-precision support for 2-bit or 3-bit groups. This quantization reduces raw weight storage by a factor of 4. Critical tensors—including norms, Hadamard diagonals, and gate tensors—remain in FP16 to preserve model accuracy. The fake_quant() helper simulates quantization during training to maintain convergence before the final export.

Stage 2: Lloyd-Max Codebook Generation

Rather than storing full per-tensor scaling factors, Needle 2 builds a Lloyd-Max codebook per group (default group size of 128) using _cq_codebook_np() in needle/model/quantize.py. This codebook is computed once during export and stored in the file header. Because the same codebook is reused for every quantized tensor, the format avoids the per-tensor overhead that plagues naive quantization schemes, reducing metadata to a few kilobytes regardless of model depth.

Stage 3: LSB-First Packing and Layout

The final stage packs quantized indices into a contiguous byte stream using LSB-first nibble packing (e.g., packing two 4-bit indices per byte). In needle/model/export.py, the _pack_lsb() and _cq_pack() routines (lines 84–100) serialize the indices alongside per-group FP16 norm vectors. The complementary _cq_unpack() routine (lines 102–118) allows the C++ runtime to reconstruct weights on-the-fly with a single fast kernel, keeping RAM usage low during inference.

Inside the .cact File Format Structure

The .cact binary is a purpose-built format designed for zero-overhead loading and memory mapping. It consists of three distinct regions: a compact header, a tensor directory, and the raw tensor blobs.

Header Metadata and Shared Codebook

The file begins with a 120-byte header containing architecture metadata: vocabulary size, model dimensions, codebook length, KV-window configuration, and quantization parameters. The shared Lloyd-Max codebook is embedded immediately following this header. This design is implemented in needle/model/export.py (lines 12–18) and ensures that dequantization parameters are loaded once and reused for every tensor in the model.

Tensor Directory Layout

Following the header is a tensor directory containing one fixed-size record per tensor. Each record stores dtype, shape, file offset, size, group identifier, and bit-width—without any tensor names. The tensor order is strictly predetermined (embedding → per-layer weights → engram tables → final norm → optional heads → tokenizer), eliminating the need for string storage and parsing overhead. See the directory layout definition in needle/model/export.py (lines 33–38).

CQ Blob Storage

Weight tensors are stored as CQ blobs: packed integer indices followed by per-group FP16 norm vectors. Small non-quantized tensors (such as the final layer norm or specific bias vectors) are stored as plain FP16 or FP32 arrays. Because the codebook and norms are shared, the total file size is dominated by the packed indices, resulting in the ~14MB footprint for 45 million parameters.

Exporting and Loading Compressed Models

The needle/model/export.py module provides high-level APIs for creating and reading .cact files, while needle/model/run.py handles runtime loading.

Exporting a Checkpoint to .cact

To compress a trained checkpoint, use write_export() with the desired bit-width and group size:

from needle.model.export import write_export
from needle.model.run import load_checkpoint
from needle.model.tokenizer import get_tokenizer

# Load training checkpoint

params, config, _ = load_checkpoint("ckpt/needle-45M.ckpt")

# Export with 4-bit CQ quantization (default)

write_export(
    params,
    config,
    path="needle.cact",        # ~14MB binary

    bits=4,                    # 4-bit CQ for weight tensors

    group=128,                 # Group size for codebook

    tokenizer=get_tokenizer(config.vocab_size),
)

The function builds the 120-byte header, serializes each tensor via _cq_pack(), and writes the contiguous binary stream.

Loading at Runtime

The runtime loader reconstructs tensors on-demand, keeping memory usage at ~28MB:

from needle.model.run import load_checkpoint
from needle.model.export import read_export

# Read .cact file

metadata, tensors = read_export("needle.cact")

# metadata contains architecture fields (num_layers, d_model, etc.)

# tensors is a list of NumPy arrays in the predetermined order

The read_export() function parses the header and directory, then uses _cq_unpack() to decompress each quantized weight only when needed.

Inspecting Codebook Efficiency

You can verify the codebook sharing by inspecting the header:

from needle.model.export import read_export

meta, _ = read_export("needle.cact")
print(f"Codebook entries: {meta['codebook'].shape[0]}")   # → 48 (2+3+4 bits)

Because the codebook is stored once in the header (as constructed in _pack_cact() at lines 47–53 of needle/model/export.py), it occupies negligible space compared to the weight data.

Summary

• CQ (Cactus-Quant) INT4 reduces weight precision from FP16 to 4-bit integers, cutting storage by 75%.
• A shared Lloyd-Max codebook (computed in _cq_codebook_np()) is stored once in the 120-byte header and reused across all tensors.
• LSB-first packing (_cq_pack()) stores indices as packed nibbles/bytes with per-group FP16 norms, optimized for fast C++ decoding.
• The .cact format eliminates tensor names and uses fixed-order records, minimizing metadata overhead.
• The final 45M-parameter model occupies ~14MB on disk and runs in ~28MB of RAM, roughly 4× smaller than an FP16 equivalent.

Frequently Asked Questions

What is CQ (Cactus-Quant) quantization?

CQ quantization is a custom vector quantization scheme implemented in needle/model/quantize.py that represents weight matrices as low-bit integers (INT4, INT3, or INT2) using a shared Lloyd-Max codebook. During inference, indices are mapped back to floating-point values by multiplying codebook centroids with per-group FP16 normalization vectors, allowing the model to retain accuracy while reducing weight storage by a factor of 4 or more.

Why is the .cact file format limited to 14MB for 45M parameters?

The 14MB footprint results from the combination of 4-bit quantization (reducing the 56MB FP16 baseline to ~14MB) and an extremely compact file structure. By storing a single shared codebook in the header, omitting tensor name strings, and using LSB-first bit packing for indices, the format ensures that nearly every byte of the 14MB file stores actual parameter data rather than metadata.

How does Needle 2 keep RAM usage at ~28MB when the model is 45M parameters?

Unlike frameworks that decompress the entire model into FP16 at load time, Needle 2 keeps weights in their compressed CQ format and dequantizes them on-the-fly during the forward pass. The _cq_unpack() function in needle/model/export.py reconstructs weight blocks lazily using the shared codebook, ensuring that only the current layer’s activations and a small buffer of dequantized weights reside in memory simultaneously.

Can I adjust the compression level or bit-width?

Yes. The write_export() function accepts a bits parameter (default 4) and a group parameter (default 128) to control the quantization granularity. You can specify mixed-precision configurations (combining 2-bit, 3-bit, and 4-bit groups) by modifying the quantization configuration before export, though the default 4-bit setting provides the optimal balance for the 45M-parameter model to stay within the 14MB target.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}