Benefits of CQ2-Bit Quantization in Needle 2: Extreme Compression for Edge AI

Needle 2 leverages CQ 2-bit (Cactus-Quants) quantization to compress a 45-million-parameter model into a 14 MB binary that runs in just 28 MB of RAM, delivering 8–10× size reduction over fp16 while maintaining accuracy through Lloyd-Max optimized codebooks and Walsh-Hadamard transforms.

The cactus-compute/needle repository implements a novel quantization scheme that stores and executes transformer weights in a ternary 2-bit format. Unlike conventional 8-bit or 16-bit quantization, CQ 2-bit quantization enables tiny binary sizes, memory-mapped inference, and fast O(n log n) arithmetic without learned parameters. This architecture makes Needle 2 ideal for air-gapped environments, edge devices, and any deployment scenario where storage and RAM are constrained.

Extreme Model Compression with Ternary Codebooks

Needle 2’s export pipeline packs the entire model into a single 14 MB .cact file by representing weight tensors as CQ 2-bit integers. This achieves roughly 4× compression compared to 8-bit formats and 8–10× reduction versus standard fp16 storage.

In needle/model/export.py, the export routine (lines 62–68) groups weights and applies a 2-bit codebook with per-group L2 normalization. Each weight group is encoded using ternary values that reference a shared codebook, eliminating the need to store full-precision floats. The cact binary header (lines 19–30) encodes the bit-width metadata (bits and kv_bits fields), allowing the runtime to interpret the quantized buffer correctly without recompilation.

Minimal Runtime Memory via Memory-Mapping

A full inference session in Needle 2 consumes approximately 28 MB of RAM, achieved through aggressive quantization and zero-copy weight loading.

The runtime loads the .cact blob as a read-only memory-mapped file, as implemented in needle/model/export.py (lines 12–17). Because the quantized weights remain in their compressed 2-bit representation throughout inference, the engine avoids the memory duplication typically required for on-the-fly dequantization. Only a small KV cache is allocated in writable memory, keeping the working set minimal.

Fast Arithmetic Using Walsh-Hadamard Transforms

CQ 2-bit quantization pairs with a fixed Walsh-Hadamard matrix to accelerate matrix-vector products to O(n log n) complexity.

In needle/model/quantize.py (lines 41–45), the cq_quantize routine applies the orthogonal transform in-place on each weight group (rot = groups @ H) before normalization. During inference, this structure allows the compute kernel to use cheap lookup operations against the 2-bit codebook rather than expensive floating-point multiplications. The transform requires no learned parameters and is applied uniformly across all groups, ensuring consistent latency characteristics.

Accuracy Preservation Through Lloyd-Max Optimization

Despite aggressive compression to 2 bits, Needle 2 maintains competitive accuracy by optimizing the quantization codebook for minimal distortion.

The _cq_codebook_np utility in needle/model/quantize.py (lines 31–40) generates a Lloyd-Max optimized codebook designed for unit-sphere weight distributions. By minimizing the expected quantization error (measured via noise_scale and cq_distortion utilities), each weight is mapped to its nearest centroid with low relative error. This codebook is computed once per bit-width and reused across all groups, ensuring that the ternary representation preserves the model's semantic capabilities.

Cross-Platform Deployment and KV-Cache Control

The CQ 2-bit format enables uniform deployment across CPU, GPU, and Metal backends without platform-specific recompilation.

The .cact file header encodes both weight precision (bits) and KV-cache precision (kv_bits), allowing independent configuration of storage and activation quantization. As defined in needle/model/quantize.py (lines 61–66), the configure_deploy function supports setting kv_bits separately from the main weights. When kv_bits is set below 8, the KV cache also uses CQ 2-bit compression, further reducing memory for long-context inference while keeping weights at 2-bit precision.

How to Export and Run CQ 2-Bit Models

You can generate a CQ 2-bit model using the Needle CLI or Python API. The following examples demonstrate loading a pre-quantized model and explicitly configuring 2-bit export:

# Load a CQ 2-bit model (default exported format)

import needle
agent = needle.Needle(weights="needle2.cact", tools=[...])
response = agent.run("What is the weather in Nairobi?")
print(response["results"])

To explicitly force 2-bit quantization during export, use the --bits flag which triggers configure_deploy in the quantization pipeline:

# Export with CQ 2-bit quantization

needle build base.pkl --out needle2.cact --bits 2

The export script handles flag parsing and invokes configure_deploy to bake the 2-bit codebook into the final binary, ensuring the runtime reads weights directly without conversion.

Summary

• CQ 2-bit quantization reduces a 45M-parameter Needle 2 model to 14 MB on disk (8–10× vs fp16).
• Memory-mapped loading keeps runtime RAM at approximately 28 MB by avoiding weight decompression copies.
• Walsh-Hadamard transforms enable O(n log n) inference speed without learned parameters.
• Lloyd-Max codebooks minimize quantization distortion to preserve model accuracy.
• Independent KV-cache precision (kv_bits) allows further memory savings for long sequences.
• Uniform binary format runs on any platform (CPU, GPU, Metal) without recompilation.

Frequently Asked Questions

What does CQ stand for in Needle 2’s quantization scheme?

CQ stands for Cactus-Quants, the custom quantization format developed for the cactus-compute/needle repository. It refers specifically to the ternary 2-bit codebook compression used to store weight tensors and KV caches in a compact, fixed-point representation.

How does CQ 2-bit quantization affect model accuracy compared to fp16?

Needle 2 maintains competitive accuracy through Lloyd-Max optimization of the quantization codebook. As implemented in needle/model/quantize.py, the codebook is pre-computed to minimize distortion for unit-sphere weight distributions, ensuring each 2-bit centroid represents the nearest full-precision value with low relative error.

Can CQ 2-bit models run on hardware other than CPU?

Yes. The .cact binary format is platform-agnostic; the same file runs on CPU, GPU, or Metal backends without recompilation. The header specifies the bit-width (bits and kv_bits), allowing the C++ inference engine to select the appropriate numeric path automatically.

Is it possible to use different bit-widths for weights and the KV cache?

Yes. Needle 2 supports decoupled precision control via the kv_bits parameter in configure_deploy (needle/model/quantize.py, lines 61–66). You can keep weights at 2-bit while configuring the KV cache to higher precision (or vice versa), trading off between memory usage and context-window stability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}