# Needle Quantization Methods: CQ2-bit, CQ3-bit, CQ4-bit, and Ternary (Cactus Quants) Explained

> Explore Needle's Cactus Quants (CQ) methods: 2-bit, 3-bit, 4-bit integer, and 1.58-bit ternary quantization. Learn about Lloyd-Max codebooks and Hadamard rotation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-20

---

**Needle uses Cactus-Quants (CQ) for 2-bit, 3-bit, and 4-bit integer quantization, plus a special 1.58-bit ternary format, all implemented with Lloyd-Max codebooks and Hadamard rotation.**

The [Needle](https://github.com/cactus-compute/needle) inference engine implements a custom quantization family called **Cactus-Quants** (CQ). These methods compress transformer weights to as low as 1.58 bits per parameter while preserving model quality through per-group codebook optimization and rotational transforms. This guide examines each quantization method, its implementation in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), and how to configure mixed-precision deployments.

---

## CQ2-bit, CQ3-bit, and CQ4-bit Integer Quantization

The core CQ family provides three symmetric integer formats storing weights in 2, 3, or 4 bits per element.

### Lloyd-Max Codebook Generation

Each quantized group receives its own optimal codebook. The function `_cq_codebook_np` in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) generates this codebook using Lloyd-Max quantization:

```python

# From needle/model/quantize.py (lines 34-47)

# _cq_codebook_np creates per-group centroids optimized for the weight distribution

```

The codebook minimizes reconstruction error for the specific weight distribution in each group, unlike uniform quantization schemes.

### Hadamard Rotation

Before quantization, weights undergo a Hadamard transform via `_cq_hadamard_np`. This rotation:

- Reduces dynamic range outliers
- Improves quantization resolution
- Makes weight distributions more Gaussian

### Reconstruction Pipeline

Quantized values are reconstructed using:

1. The per-group codebook index
2. The per-group norm for rescaling
3. Inverse Hadamard rotation

The `cq_quantize` function orchestrates this full pipeline.

---

## Ternary Quantization (1.58-bit)

For extreme compression, Needle supports a fixed ternary format with exactly three levels.

**Ternary codebook values:** `[-1.2240064, 0, 1.2240064]`

This format activates automatically when `bits == TERNARY_BITS` (defined as 1.58). Unlike adaptive CQ formats, the ternary codebook is fixed globally in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 8-12):

```python

# needle/model/quantize.py

TERNARY_BITS = 1.58
_TERNARY_CB = jnp.array([-1.2240064, 0.0, 1.2240064])

```

Ternary quantization eliminates the need to store codebooks, reducing memory overhead further.

---

## Quantization-Aware Training (QAT) with Fake-Quant

Needle simulates quantization effects during training without actually compressing tensors. The **fake-quant** mechanism lets gradients flow through full-precision weights while the forward pass sees quantized values.

### How Fake-Quant Works

The `fake_quant` function adds a stop-gradient term:

```python

# q is the quantized weight, w is the full-precision weight

output = w + stop_gradient(q - w)  # Forward sees q, backward sees w

```

### Training Integration

Two functions manage QAT lifecycle in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 10-22):

- `configure_qat(every, weight_group, weight_bits)` — schedules quantization frequency
- `maybe_quant_weights` — applies fake-quant to parameters when enabled

Example training configuration:

```python
from needle.model.quantize import configure_qat, quantize_params_configured

# Enable fake-quant every 1000 steps with 4-bit CQ

configure_qat(every=1000, weight_group=128, weight_bits=4)

# Apply to Flax parameter tree

params_qat = quantize_params_configured(params)

```

---

## Mixed-Precision Per-Tensor Quantization

Different layers benefit from different bit-widths. Needle's **bits map** system assigns custom precision per tensor.

### Parsing Bits Maps

The `parse_bits_map` function interprets configuration strings:

```python
from needle.model.quantize import parse_bits_map

bits_map, default = parse_bits_map("default=4,attn.q_proj=2")

# Result: 4-bit everywhere except 2-bit for attention query projections

```

Valid bit-widths: 2, 3, 4, or 1.58 (ternary).

### Applying Mixed Precision

`cq_mixed_params` looks up each parameter's assigned width and dispatches to `cq_quantize`:

```python
from needle.model.quantize import cq_mixed_params

params_cq = cq_mixed_params(params, bits_map, default_bit_width)

```

This enables aggressive compression for attention layers while preserving capacity in feed-forward networks.

---

## W4A8 Export Format

Deployed models use a standard binary layout documented in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 5-7):

> "W4A8: the matmul weights are Cactus-Quants INT4"

| Component | Format |
|-----------|--------|
| Matmul weights | CQ INT4 (4-bit) |
| Activations | 8-bit |
| Norm/gate weights | FP16 |

The [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py) module handles packing quantized codebooks, indices, and norms into the on-disk representation.

---

## Summary

- **Cactus-Quants (CQ)** provide 2-bit, 3-bit, and 4-bit formats using Lloyd-Max codebooks and Hadamard rotation
- **Ternary quantization** offers 1.58-bit compression with a fixed three-level codebook
- **Fake-quant QAT** simulates quantization during training without compressing gradients
- **Mixed-precision** maps different bit-widths to different tensors via parseable configuration strings
- **W4A8 export** stores weights as CQ INT4 for deployment, with 8-bit activations

---

## Frequently Asked Questions

### What makes Cactus-Quants different from GPTQ or AWQ?

Cactus-Quants use per-group Lloyd-Max codebooks rather than per-channel or per-tensor scaling. The Hadamard rotation step, implemented in `_cq_hadamard_np`, is specifically designed to handle outliers before quantization. This combination targets transformer weight distributions more aggressively than uniform quantization approaches.

### Can I use ternary quantization for all layers?

Yes, by setting `bits=1.58` or `TERNARY_BITS` in your configuration. However, the fixed `[-1.2240064, 0, 1.2240064]` ternary codebook provides coarser approximation than adaptive CQ formats. Mixed-precision configurations typically assign ternary only to selected attention projections rather than the full model.

### How does fake-quant affect training convergence?

Fake-quant adds quantization noise to the forward pass while preserving gradient precision. The `every` parameter in `configure_qat` controls frequency—intermittent application reduces training instability. Gradients always flow through uncompressed weights, so optimizer states remain full-precision.

### What group sizes work best with CQ quantization?

The `weight_group` parameter in `configure_qat` typically ranges from 64 to 256. Smaller groups improve reconstruction accuracy but increase codebook overhead. The default 128 provides a standard tradeoff, though layer-specific tuning via `parse_bits_map` can optimize for your target memory budget.