How CQ2-bit Quantization Contributes to Needle 2's Small Footprint

CQ2-bit quantization enables Needle 2 to compress a 45-million-parameter model into a single 14 MiB binary that runs full sessions in approximately 28 MiB of RAM.

Needle 2, developed by Cactus Compute, achieves its remarkable efficiency through a custom blockwise quantization scheme called Cactus-Quants (CQ). This article examines the technical mechanisms behind CQ2-bit quantization and how they deliver a 6× size reduction compared to full-precision storage.

What is CQ2-bit Quantization?

CQ2-bit quantization is a group-wise weight compression technique that stores neural network parameters using only 2 bits per weight instead of the standard 16-bit floating-point format. The implementation lives in needle/model/quantize.py, which defines the core cq_quantize function and supporting infrastructure.

The quantizer supports multiple bit widths through the CB_BITS = (2, 3, 4) constant, but Needle 2 specifically targets 2-bit compression for maximum deployment efficiency.

Blockwise Quantization Architecture

The CQ quantizer operates on groups of 128 weights, applying a sequence of deterministic transformations that preserve model quality while minimizing storage.

Hadamard Rotation

Each weight group undergoes rotation by a fixed Hadamard matrix (_cq_hadamard_np), an orthonormal transform implemented in lines 119-124 of quantize.py:

def cq_quantize(w, bits, group_size=128, codebook=None):
    cb = codebook if codebook is not None else jnp.asarray(_cq_codebook_np(bits, group_size))
    # Pad to multiple of group_size → reshape → Hadamard transform → quantize → inverse transform

    # ...

    return deq[..., :D] if pad else deq

The Hadamard matrix is analytically defined and parameter-free. This means zero storage overhead for the transform itself—a critical advantage over learned quantization methods.

Static Codebook Quantization

After rotation, each component is quantized to the nearest entry in a precomputed codebook. For 2-bit quantization, the codebook contains exactly 4 entries.

The 2-bit codebook is generated by Lloyd-Max clustering of a Gaussian distribution (_lloyd_max_gaussian in lines 95-106). This deterministic construction means:

• The same codebook is reused across all layers
• No per-layer codebook storage is required
• Inference remains bit-exact across platforms

Per-Group Storage Breakdown

CQ2-bit quantization achieves compression through a careful separation of structural and scale information. For each group of 128 weights:

• 2 bits per weight for the codebook index (the structural information)
• 16-bit float for the per-group L2 norm (the scale information)
• No storage for the Hadamard matrix (implicit, fixed transform)

This produces the exact formula implemented in cq_model_bytes (lines 75-88):

# Estimate size after quantization

from needle.model.quantize import cq_model_bytes

size_bytes = cq_model_bytes(quantized_params, bits=2)

For a 45-million-parameter model, this yields approximately 14 MiB—matching the size advertised in the README—versus ~90 MiB for FP16 storage.

Integration with the Export Pipeline

The quantizer connects to deployment through two key integration points in needle/model/export.py and quantize.py.

Configuration and Quantization

The configure_deploy function sets _WEIGHT_BITS = 2 (lines 56-63), which quantize_params_configured uses during export:

from needle.model.quantize import configure_deploy, quantize_params_configured

# Configure for 2-bit weight quantization

configure_deploy(act_bits=8, kv_bits=0, kv_group=64)
params_q2 = quantize_params_configured(original_params)

Binary Serialization

The export() function in export.py (lines 51-58) writes the final .cact blob containing:

1. Packed 2-bit indices (consecutive weights packed into bytes)
2. Per-group FP16 norms (appended after the index data)

from needle.model.export import export

export(
    params=params_q2,
    config=my_config,
    out_path="needle2_2bit.cact",
    bits="2"
)

At runtime, needle/_worker.py loads this compact blob, with the small size directly reducing RAM pressure during inference.

Architectural Benefits of CQ2-bit Design

Beyond raw compression, CQ2-bit quantization provides structural advantages that align with Needle 2's design goals.

Zero parameter overhead. The Hadamard transform requires no learned or stored parameters, avoiding the "compression tax" common in other quantization schemes.

Fast O(N log N) operations. Quantization and dequantization reduce to matrix multiplications with the Hadamard matrix, which can be implemented efficiently using Fast Walsh-Hadamard Transform techniques.

Deterministic cross-platform inference. Static codebooks eliminate non-determinism from learned quantization tables, ensuring identical outputs across deployment targets.

Efficient memory bandwidth. The 2-bit representation reduces weight memory traffic by 8× during inference, critical for edge devices with constrained memory subsystems.

Summary

• CQ2-bit quantization compresses Needle 2's 45M parameters into 14 MiB by storing 2 bits per weight plus minimal per-group metadata
• The needle/model/quantize.py implementation uses Hadamard rotation to spread energy uniformly before static codebook quantization
• Per-group storage includes 2-bit indices and a 16-bit FP16 norm—no other overhead
• The cq_model_bytes function and export pipeline in needle/model/export.py ensure accurate size prediction and efficient serialization
• These mechanisms combine to deliver 6× compression with preserved model capability, enabling Needle 2's tiny deployment footprint

Frequently Asked Questions

What does CQ stand for in CQ2-bit quantization?

CQ stands for Cactus-Quants, the custom quantization framework developed by Cactus Compute specifically for the Needle model family. It is not a general industry standard like INT8 or FP4, but rather an optimized blockwise scheme designed for small language model deployment.

Why 2 bits specifically rather than 4 or 8 bits?

Needle 2 targets extreme edge deployment where binary size and RAM footprint are the primary constraints. While CQ supports 2, 3, and 4 bits (via CB_BITS), the 2-bit mode achieves the critical 14 MiB threshold that enables single-file distribution. The Hadamard rotation makes 2-bit quantization viable by ensuring energy is distributed evenly across components, minimizing quantization error.

How does CQ2-bit quantization affect inference speed?

Dequantization overhead is minimal due to the Fast Walsh-Hadamard Transform structure. The rot @ H operation runs in O(N log N) time, and the 8× reduction in memory bandwidth often yields net speedup on memory-bound inference workloads. The fixed codebook also eliminates lookup table cache misses compared to learned codebook approaches.

Can CQ2-bit quantization be applied to other models?

The CQ quantizer in needle/model/quantize.py is implemented generally for JAX parameter dictionaries and could theoretically apply to other architectures. However, the 2-bit mode is specifically tuned and validated for Needle 2's "Simple Attention Network" architecture. Applying it to other models would require recalibration of the quantization-aware training or post-training quantization pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}