CQ2-Bit Weight Compression in Needle 2: How Cactus Quants Achieves 14MB Model Size

CQ2-bit weight compression is a custom quantization technique called Cactus Quants (CQ) that packs Needle 2's weight matrices into 2 bits per weight using Hadamard rotation and Lloyd-Max codebook optimization, enabling the 45M-parameter model to fit in a single 14MB binary.

Needle 2, the inference engine from the cactus-compute/needle repository, achieves its remarkably small footprint through aggressive weight compression. The system stores its entire 45-million-parameter model in just 14 megabytes by employing a proprietary quantization scheme known as Cactus Quants (CQ). This technique reduces storage to roughly one-quarter of an FP16 baseline while maintaining inference quality.

What Is CQ2-Bit Weight Compression?

CQ2-bit refers to the specific configuration of Cactus Quants that quantizes model weights to 2 bits per weight. Unlike standard 8-bit or 16-bit floating-point formats, this custom codec processes weights in fixed-size groups and applies mathematical rotation to maximize quantization efficiency. The compression is lossy but optimized to preserve the model's ability to perform tool-calling and structured extraction tasks.

The implementation centers on a group size of 128 values, defined by the constant CQ_GROUP_SIZE = 128 in the source. For each group, the algorithm applies a Hadamard transform, maps the rotated values to a compact codebook, then reverses the rotation to produce the final compressed representation.

How the CQ2-Bit Compression Pipeline Works

The compression logic resides in needle/model/quantize.py and follows a five-stage pipeline:

Grouping and Padding

The algorithm first ensures tensor dimensions align with the group size. The code calculates padding using pad = (-D) % g where D is the dimension and g is the group size (128). If padding is required, the system applies jnp.pad(w, ...) to extend the tensor, creating uniform groups for processing.

Hadamard Rotation

Each group undergoes rotation via a normalized Hadamard matrix generated by _cq_hadamard_np. The operation rot = groups @ H transforms the weight distribution to reduce dynamic range before quantization. This rotation step is critical for concentrating values around zero, which improves the accuracy of low-bit encoding.

Codebook Selection and Quantization

The system selects a codebook using _cq_codebook_np(bits, group_size), where bits = 2 for CQ2-bit mode. For 2-bit quantization, the codebook is generated either through ternary optimization or Lloyd-Max Gaussian clustering (_lloyd_max_gaussian). The quantization step uses nearest-neighbor mapping: vectors are normalized with unit = rot / jnp.maximum(norm, 1e-12), then snapped to the closest codebook entry via _cq_nearest(unit, cb).

De-rotation and Trimming

After quantization, the values pass through the inverse Hadamard transform ( ... ) @ H to restore original orientation. The system then removes any padding added during the grouping phase using return deq[..., :D] if pad else deq, yielding the final compressed weights.

Implementation in the Needle 2 Codebase

The CQ2-bit system is orchestrated through specific APIs and configuration files:

File	Role
needle/model/quantize.py	Core implementation containing cq_quantize, cq_quantize_params, and rotation matrices
needle/model/export.py	Handles model export with weight_bits specification and constructs the "CQ W2" label
needle/_worker.py	Runtime loader (needle_load) that streams and decompresses weights on-demand

To enable CQ2-bit during deployment, the code constructs a configuration object with weight_bits="2" and passes it to deploy_quantize. The function signature accepts parameters alongside this config: deploy_quantize(params, config=...). The default quantization settings are established via configure_deploy(act_bits=8, kv_bits=8, kv_group=64), which prepares the quantization context before the 2-bit weight conversion occurs.

Enabling CQ2-Bit Compression in Practice

To load a pre-compressed Needle 2 model using the CQ2-bit binary format:

import needle

# Load a .cact file exported with weight_bits="2"

agent = needle.Needle(
    tools="[]",
    weights="acme/needle-2.cact"  # CQ2-bit compressed binary

)

result = agent("Summarize the last news headlines.")
print(result)

To manuallyquantize a model using the CQ2-bit pipeline during export:

from needle.model.quantize import configure_deploy, deploy_quantize

# Configure activation and KV cache quantization

configure_deploy(act_bits=8, kv_bits=8, kv_group=64)

# Apply CQ2-bit weight compression

quantized_params, label = deploy_quantize(
    params,
    config=type("Cfg", (), {"weight_bits": "2"})()  # Trigger CQ2-bit

)

print(label)  # Output: "CQ W2"

# Save quantized_params to a .cact file for distribution

Memory and Performance Benefits

The CQ2-bit format delivers substantial efficiency gains:

• Storage Footprint: Reduces the 45M-parameter model to 14MB, compared to approximately 90MB for FP16 weights (a ~6x reduction).
• Runtime RAM: The engine requires only ~28MB of RAM during inference because weights are streamed and decompressed on-the-fly via needle_load in needle/_worker.py.
• Latency: Hadamard transforms and small codebook lookups operate efficiently on modest hardware, with dequantization happening lazily during token generation.

Summary

• CQ2-bit stands for Cactus Quants 2-bit, a custom quantization method using 2 bits per weight.
• The algorithm processes weights in groups of 128, applying Hadamard rotation and Lloyd-Max codebook optimization.
• Implementation lives in needle/model/quantize.py, with APIs exposed through configure_deploy and deploy_quantize.
• 45 million parameters compress into 14MB, requiring only 28MB RAM at runtime.
• Compression is triggered by setting weight_bits="2" in the deployment configuration.

Frequently Asked Questions

What does CQ stand for in Needle 2's compression system?

CQ stands for Cactus Quants, the proprietary quantization framework developed for the Needle inference engine. It encompasses multiple bit-width configurations, with CQ2-bit specifically referring to the 2-bit-per-weight variant used in Needle 2's smallest deployment targets.

How does CQ2-bit compression achieve 2 bits per weight without significant accuracy loss?

The system uses Hadamard rotation to redistribute weight values before quantization, making them more amenable to low-bit encoding. After rotation, a Lloyd-Max optimized codebook (or ternary codebook) represents the values using only four possible states (2 bits). The de-rotation step restores the original vector orientations, minimizing reconstruction error during inference.

What is the file size reduction compared to standard FP16 weights?

Needle 2's 45-million-parameter model occupies 14MB with CQ2-bit compression, compared to approximately 90MB in FP16 format. This represents an 84% reduction in storage size, or roughly one-sixth of the original footprint, while maintaining functional capability for tool-calling and structured generation tasks.

Which API functions control CQ2-bit compression during model export?

The compression pipeline is controlled by two primary functions from needle.model.quantize: configure_deploy() sets the global quantization context for activations and KV caches, while deploy_quantize() applies the specific weight compression when passed a configuration object with weight_bits="2". The resulting quantized parameters are then serialized through the export pipeline in needle/model/export.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}