Needle Quantization Methods: CQ2-bit, CQ3-bit, CQ4-bit, and Ternary (Cactus Quants) Explained

Needle uses Cactus-Quants (CQ) for 2-bit, 3-bit, and 4-bit integer quantization, plus a special 1.58-bit ternary format, all implemented with Lloyd-Max codebooks and Hadamard rotation.

The Needle inference engine implements a custom quantization family called Cactus-Quants (CQ). These methods compress transformer weights to as low as 1.58 bits per parameter while preserving model quality through per-group codebook optimization and rotational transforms. This guide examines each quantization method, its implementation in needle/model/quantize.py, and how to configure mixed-precision deployments.

---

CQ2-bit, CQ3-bit, and CQ4-bit Integer Quantization

The core CQ family provides three symmetric integer formats storing weights in 2, 3, or 4 bits per element.

Lloyd-Max Codebook Generation

Each quantized group receives its own optimal codebook. The function _cq_codebook_np in needle/model/quantize.py generates this codebook using Lloyd-Max quantization:

# From needle/model/quantize.py (lines 34-47)

# _cq_codebook_np creates per-group centroids optimized for the weight distribution

The codebook minimizes reconstruction error for the specific weight distribution in each group, unlike uniform quantization schemes.

Hadamard Rotation

Before quantization, weights undergo a Hadamard transform via _cq_hadamard_np. This rotation:

• Reduces dynamic range outliers
• Improves quantization resolution
• Makes weight distributions more Gaussian

Reconstruction Pipeline

Quantized values are reconstructed using:

1. The per-group codebook index
2. The per-group norm for rescaling
3. Inverse Hadamard rotation

The cq_quantize function orchestrates this full pipeline.

---

Ternary Quantization (1.58-bit)

For extreme compression, Needle supports a fixed ternary format with exactly three levels.

Ternary codebook values: [-1.2240064, 0, 1.2240064]

This format activates automatically when bits == TERNARY_BITS (defined as 1.58). Unlike adaptive CQ formats, the ternary codebook is fixed globally in needle/model/quantize.py (lines 8-12):

# needle/model/quantize.py

TERNARY_BITS = 1.58
_TERNARY_CB = jnp.array([-1.2240064, 0.0, 1.2240064])

Ternary quantization eliminates the need to store codebooks, reducing memory overhead further.

---

Quantization-Aware Training (QAT) with Fake-Quant

Needle simulates quantization effects during training without actually compressing tensors. The fake-quant mechanism lets gradients flow through full-precision weights while the forward pass sees quantized values.

How Fake-Quant Works

The fake_quant function adds a stop-gradient term:

# q is the quantized weight, w is the full-precision weight

output = w + stop_gradient(q - w)  # Forward sees q, backward sees w

Training Integration

Two functions manage QAT lifecycle in needle/model/quantize.py (lines 10-22):

• configure_qat(every, weight_group, weight_bits) — schedules quantization frequency
• maybe_quant_weights — applies fake-quant to parameters when enabled

Example training configuration:

from needle.model.quantize import configure_qat, quantize_params_configured

# Enable fake-quant every 1000 steps with 4-bit CQ

configure_qat(every=1000, weight_group=128, weight_bits=4)

# Apply to Flax parameter tree

params_qat = quantize_params_configured(params)

---

Mixed-Precision Per-Tensor Quantization

Different layers benefit from different bit-widths. Needle's bits map system assigns custom precision per tensor.

Parsing Bits Maps

The parse_bits_map function interprets configuration strings:

from needle.model.quantize import parse_bits_map

bits_map, default = parse_bits_map("default=4,attn.q_proj=2")

# Result: 4-bit everywhere except 2-bit for attention query projections

Valid bit-widths: 2, 3, 4, or 1.58 (ternary).

Applying Mixed Precision

cq_mixed_params looks up each parameter's assigned width and dispatches to cq_quantize:

from needle.model.quantize import cq_mixed_params

params_cq = cq_mixed_params(params, bits_map, default_bit_width)

This enables aggressive compression for attention layers while preserving capacity in feed-forward networks.

---

W4A8 Export Format

Deployed models use a standard binary layout documented in needle/model/export.py (lines 5-7):

"W4A8: the matmul weights are Cactus-Quants INT4"

Component	Format
Matmul weights	CQ INT4 (4-bit)
Activations	8-bit
Norm/gate weights	FP16

The export.py module handles packing quantized codebooks, indices, and norms into the on-disk representation.

---

Summary

• Cactus-Quants (CQ) provide 2-bit, 3-bit, and 4-bit formats using Lloyd-Max codebooks and Hadamard rotation
• Ternary quantization offers 1.58-bit compression with a fixed three-level codebook
• Fake-quant QAT simulates quantization during training without compressing gradients
• Mixed-precision maps different bit-widths to different tensors via parseable configuration strings
• W4A8 export stores weights as CQ INT4 for deployment, with 8-bit activations

---

Frequently Asked Questions

What makes Cactus-Quants different from GPTQ or AWQ?

Cactus-Quants use per-group Lloyd-Max codebooks rather than per-channel or per-tensor scaling. The Hadamard rotation step, implemented in _cq_hadamard_np, is specifically designed to handle outliers before quantization. This combination targets transformer weight distributions more aggressively than uniform quantization approaches.

Can I use ternary quantization for all layers?

Yes, by setting bits=1.58 or TERNARY_BITS in your configuration. However, the fixed [-1.2240064, 0, 1.2240064] ternary codebook provides coarser approximation than adaptive CQ formats. Mixed-precision configurations typically assign ternary only to selected attention projections rather than the full model.

How does fake-quant affect training convergence?

Fake-quant adds quantization noise to the forward pass while preserving gradient precision. The every parameter in configure_qat controls frequency—intermittent application reduces training instability. Gradients always flow through uncompressed weights, so optimizer states remain full-precision.

What group sizes work best with CQ quantization?

The weight_group parameter in configure_qat typically ranges from 64 to 256. Smaller groups improve reconstruction accuracy but increase codebook overhead. The default 128 provides a standard tradeoff, though layer-specific tuning via parse_bits_map can optimize for your target memory budget.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}