How CQ2-bit Quantization Works with Cactus Quants in Needle 2

CQ2-bit quantization in Needle 2 leverages Cactus Quants (CQ) to compress neural network weights to 2-bit precision using Hadamard rotation, Lloyd-Max optimized codebooks, and group-wise normalization, achieving a 4× reduction in memory bandwidth without altering forward-pass semantics.

Needle 2 implements an advanced weight quantization system called "Cactus Quants" that enables extreme compression down to 2 bits per weight. This technique, located in the cactus-compute/needle repository, combines signal processing principles with efficient JAX implementations to reduce model memory footprints while maintaining computational accuracy.

The Core Mechanism of Cactus Quants

The CQ2-bit quantization algorithm follows a four-stage pipeline that processes weight tensors in fixed-size groups. When bits == 2, the system constructs a 4-level quantization codebook and applies rotational transforms to improve quantization efficiency.

Lloyd-Max Codebook Generation

For 2-bit precision, Needle 2 pre-computes optimal quantization levels using a Lloyd-Max Gaussian optimizer. In needle/model/quantize.py, the function _cq_codebook_np invokes _lloyd_max_gaussian to generate a 4-entry codebook (2² levels) scaled according to the group dimensions. This codebook is computed once and cached, ensuring deterministic quantization across inference batches.

Hadamard Rotation and Group Processing

Weight tensors are partitioned into blocks of size group_size (defaulting to 128). Each block undergoes rotation using a Hadamard matrix H generated by _cq_hadamard_np. The rotation rot = groups @ H decorrelates weight values and uniformly distributes energy across the tensor, which significantly improves the signal-to-noise ratio of subsequent scalar quantization.

Normalization and Nearest-Codebook Lookup

After rotation, each block is normalized to unit length using norm = sqrt(sum(rot²)). These normalization factors are preserved in float16 to minimize storage overhead while retaining sufficient precision. The function _cq_nearest performs vector quantization by mapping each normalized value to the closest entry in the pre-computed 4-level codebook. The quantized values are then rescaled by norm before inverse transformation.

Implementation Details in needle/model/quantize.py

The practical implementation centers on needle/model/quantize.py, which exposes high-level APIs for integrating CQ quantization into model architectures.

The cq_quantize Entry Point

The public function cq_quantize serves as the primary interface for CQ2-bit quantization. It orchestrates the rotation, normalization, codebook lookup, and inverse rotation workflow, returning a tensor that appears identical to the original floating-point tensor during forward passes while internally storing only 2 bits per weight. The implementation wraps critical operations with jax.lax.stop_gradient to prevent quantization from affecting gradient flow during quantization-aware training.

Model Integration and KV-Cache Quantization

Needle 2 integrates CQ quantization through helper functions like cq_quantize_params, which traverses model parameter PyTrees to apply cq_quantize recursively. For transformer architectures, maybe_quant_kv conditionally applies 2-bit quantization to KV-cache tensors, enabling aggressive compression of attention key-value stores without manual intervention.

Runtime Behavior and Memory Efficiency

During inference, Needle 2 de-quantizes tensors on-the-fly using the stored Hadamard matrices and normalization factors. This approach preserves exact floating-point computation semantics while reducing memory bandwidth requirements by a factor of 4 (compressing 8-bit values to 2-bit representations). The de-quantization overhead is minimized through JAX's JIT compilation and optimized matrix operations.

Practical Example: Quantizing a Linear Layer

The following example demonstrates applying CQ2-bit quantization to a transformer linear layer kernel:

import jax
import jax.numpy as jnp
from needle.model.quantize import cq_quantize, configure_deploy

# Configure deployment for weight-only quantization

configure_deploy(act_bits=8, kv_bits=0)

# Initialize a weight matrix (e.g., 768 × 3072 kernel)

W = jax.random.normal(jax.random.PRNGKey(0), (768, 3072))

# Apply CQ 2-bit quantization with default group size 128

W_q = cq_quantize(W, bits=2)

# Use in forward pass with identical semantics to full precision

def linear(x):
    return x @ W_q.T

x = jax.random.normal(jax.random.PRNGKey(1), (1, 768))
y = linear(x)  # Executes with 4× memory bandwidth reduction

Summary

• CQ2-bit quantization uses a 4-level Lloyd-Max optimized codebook to represent weights with only 2 bits per parameter.
• Hadamard rotation in needle/model/quantize.py decorrelates weight groups of size 128, improving quantization efficiency through energy spreading.
• Runtime de-quantization occurs on-the-fly during inference, preserving exact forward-pass semantics while reducing memory bandwidth by 4×.
• Integration APIs like cq_quantize and cq_quantize_params enable seamless application to model parameters and KV caches with automatic gradient blocking for training stability.

Frequently Asked Questions

What is the default group size for CQ2-bit quantization in Needle 2?

The default group size is 128, as defined in the Cactus Quants implementation. This means weight tensors are partitioned into blocks of 128 elements before applying Hadamard rotation and quantization. This size balances quantization accuracy with computational overhead.

How does Needle 2 preserve floating-point semantics with 2-bit weights?

Needle 2 preserves semantics through on-the-fly de-quantization. During the forward pass, the cq_quantize function returns a de-quantized view obtained by inverse Hadamard rotation (deq = (nearest * norm) @ H). The underlying storage uses 2-bit codes, but the computation uses reconstructed float values, ensuring identical mathematical results to unquantized layers while reducing memory transfer costs.

Can CQ quantization be applied to the KV cache in transformer models?

Yes, Needle 2 supports KV-cache quantization through the maybe_quant_kv function. This helper conditionally applies CQ2-bit quantization to attention key and value tensors, significantly reducing memory consumption during long-context inference. The configuration is controlled via configure_deploy(kv_bits=2) or similar settings.

What is the memory bandwidth reduction when using CQ2-bit quantization?

CQ2-bit quantization achieves a 4× reduction in memory bandwidth compared to 8-bit representations, and a 16× reduction compared to 32-bit floating point. Since weights are stored as 2-bit codes and de-quantized near the compute units, the primary memory traffic moves 4× less data than standard 8-bit quantization schemes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}