CQ2-Bit Quantization in Needle 2: Compressing 45M Parameters into 14MB

CQ2-bit quantization is the Cactus Quants (CQ) group-wise compression scheme that reduces Needle 2's 45 million parameters to a 14MB binary using 2-bit codebooks, Hadamard transforms, and stochastic rounding, enabling inference on devices with approximately 28MB of RAM.

Needle 2 from the cactus-compute/needle repository achieves extreme model compression through CQ2-bit quantization, a technique that quantizes each weight to just two bits while preserving the performance of the full-precision model. By implementing group-wise processing with orthogonal rotations and learned codebooks in needle/model/quantize.py, this method produces compact .cact binaries that require only 2 bits per weight plus minimal overhead. The result is a deployable model that maintains tool-calling capabilities while operating within severe memory constraints typical of microcontrollers and edge devices.

How CQ2-Bit Quantization Works

The CQ2-bit scheme combines four technical components to minimize distortion when compressing 32-bit floats into 2-bit representations.

Group-Wise Quantization

The algorithm splits each weight tensor into fixed-size groups (default group size of 128) and quantizes each group independently. Processing smaller groups keeps quantization error localized and manageable, allowing coarse 2-bit representations to approximate high-precision values with minimal accuracy loss compared to full-tensor quantization.

Hadamard Transform

Before quantization, each group undergoes a Walsh-Hadamard transform using an orthogonal matrix H. This rotation spreads weight information uniformly across the vector, reducing the dynamic range and ensuring that a simple 4-level 2-bit codebook can effectively represent the transformed values. The Hadamard transform is critical for making ultra-low-bit quantization feasible without significant performance degradation.

Learned Codebooks

Quantization maps transformed weights to discrete values using static lookup tables built offline. The codebooks are constructed via the Lloyd-Max algorithm (k-means) on Gaussian-distributed data, generating four centroids for the 2-bit case. During inference, each weight is replaced by its nearest centroid index (2 bits), then rescaled by the group's norm to reconstruct the approximate value.

Stochastic Rounding and Straight-Through Estimator

Training stability relies on the cq_ste (straight-through estimator) and add_cq_noise functions. During forward passes, the stochastic rounding process applies the quantizer but allows gradients to flow through the original high-precision weights. Additionally, calibrated Gaussian noise injection before quantization improves the model's tolerance to the aggressive compression, ensuring the learned weights remain robust when deployed at 2-bit precision.

Needle 2 Implementation Pipeline

The export pipeline in needle/model/export.py orchestrates the conversion from full-precision checkpoints to compressed binaries through three stages.

Configuration

Users specify target precision via configure_deploy() in needle/model/quantize.py. Setting act_bits=2, kv_bits=2, and kv_group=64 prepares the system for 2-bit activation and key-value cache quantization:

from needle.model.quantize import configure_deploy

configure_deploy(act_bits=2, kv_bits=2, kv_group=64)

Parameter Quantization

The cq_quantize_params() function iterates over all weight tensors, applying the Hadamard rotation, looking up nearest codebook entries, and rescaling by group norms. This function accepts bits=2 and group_size=128 parameters to control the compression level:

from needle.model.quantize import cq_quantize_params

quantized_params = cq_quantize_params(
    base.model.params, 
    bits=2, 
    group_size=128
)

Binary Export

The export_cact() function writes the quantized parameters to disk, while the CLI command needle build --bits 2 triggers the full export sequence. According to the README.md, this produces a .cact binary containing all quantized weights and scaling factors required by the inference engine.

Practical Usage Example

The following workflow demonstrates loading a base checkpoint, configuring 2-bit quantization, and exporting the compressed model:

import needle
from needle.model.quantize import configure_deploy, cq_quantize_params
from needle.model.export import export_cact

# Load base checkpoint

base = needle.Needle(weights="cactus-needle/needle2")

# Configure for 2-bit deployment

configure_deploy(act_bits=2, kv_bits=2, kv_group=64)

# Quantize parameters (typically invoked via CLI)

quantized_params = cq_quantize_params(
    base.model.params, 
    bits=2, 
    group_size=128
)

# Export to .cact binary

export_cact(quantized_params, out_path="my_needle_2bit.cact")

Summary

• CQ2-bit quantization compresses Needle 2's 45M parameters into a 14MB binary suitable for ~28MB RAM environments.
• The technique uses group-wise processing (groups of 128) with Hadamard transforms to prepare weights for aggressive compression.
• Learned codebooks generated via the Lloyd-Max algorithm map values to 2-bit indices (4 levels) with minimal distortion.
• cq_ste and noise injection enable training stability despite ultra-low precision.
• The implementation resides primarily in needle/model/quantize.py, with export logic in needle/model/export.py and API documentation in doc/apis.md.

Frequently Asked Questions

How much memory does CQ2-bit quantization save?

CQ2-bit quantization reduces the model from its full-precision size to a 14MB binary that requires approximately 28MB of RAM during inference. This represents roughly a 10x or greater compression ratio compared to standard 32-bit floating-point representations, enabling deployment on microcontrollers and other memory-constrained devices.

What is the Hadamard transform's role in CQ2-bit quantization?

The Hadamard transform rotates weight groups using an orthogonal matrix before quantization, distributing information uniformly across the vector components. This preprocessing step reduces the effective dynamic range of the data, allowing a simple 4-entry 2-bit codebook to accurately approximate the original values without significant loss of model capability.

How does Needle 2 maintain accuracy with only 2 bits per weight?

Accuracy is preserved through group-wise quantization (keeping groups small at 128 elements), learned Lloyd-Max codebooks optimized for the weight distribution, and noise injection during training that improves robustness to quantization error. The straight-through estimator (cq_ste) further ensures gradients flow properly during backpropagation, allowing the network to learn weights that perform well when compressed.

What file format stores the quantized model?

Needle 2 saves CQ2-bit quantized models as .cact files, a custom binary format generated by export_cact() in needle/model/export.py. These files contain the quantized weight indices, group scaling factors, and codebook references necessary for the inference engine to reconstruct activations during model execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}