# Cactus Quants W4A8 Quantization in Needle 2: A Complete Technical Guide

> Explore Cactus Quants W4A8 quantization in Needle 2. Learn how this scheme compresses model weights to 4-bit precision while preserving 8-bit activations. Discover the technical details of Hadamard rotation and Lloyd-Max optimi...

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-30

---

**The Cactus Quants (CQ) W4A8 scheme compresses model weights to 4-bit precision while maintaining 8-bit activations through group-wise Hadamard rotation and Lloyd-Max Gaussian codebook optimization.**

The **Cactus Quants W4A8 quantization** format serves as the default deployment configuration in Needle 2, the open-source inference engine developed by Cactus Compute. This hybrid precision approach reduces memory bandwidth pressure by storing weights at 4-bit resolution while preserving full 8-bit arithmetic for activations, striking an optimal balance between compression and hardware efficiency on modern accelerators.

## What Is CQ W4A8 Quantization?

**CQ W4A8** is a mixed-precision quantization specification where **W4** denotes 4-bit weights and **A8** denotes 8-bit activations. Unlike uniform quantization that applies the same bit-width across all tensors, the Cactus Quants implementation in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) employs sophisticated techniques to minimize precision loss at ultra-low bit-widths.

The scheme targets the memory bottleneck in transformer inference. By compressing weights to 4 bits (achieving approximately 50% size reduction compared to 8-bit formats), the system reduces data movement costs while keeping activations at 8 bits to maintain matmul efficiency on standard GPU tensor cores.

## How Weight Quantization Works in Needle 2

The core weight quantization logic resides in the `cq_quantize` function within [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 34-47). This implementation uses a combination of group-wise processing and optimal codebook generation rather than simple linear quantization.

### Group-Wise Hadamard Rotation

Before quantization, weight tensors are divided into groups of **128 elements** (controlled by `CQ_GROUP_SIZE = 128`). Each group undergoes a **Hadamard rotation** using the normalized Hadamard matrix cached in `_cq_hadamard_np` (lines 20-24).

The rotation spreads information uniformly across the group vector, decorrelating weight magnitudes and improving the signal-to-noise ratio when subsequently mapped to discrete centroids. This preprocessing step is critical for maintaining model accuracy at 4-bit precision.

### Lloyd-Max Gaussian Codebook Generation

For 4-bit quantization, Needle generates a codebook of **16 optimal levels** (2^4) using the `_lloyd_max_gaussian` optimizer. This algorithm computes centroid boundaries that minimize mean squared error under a Gaussian distribution assumption, yielding superior reconstruction quality compared to uniform quantization grids.

The resulting codebook is cached in `_cq_codebook_np` upon first invocation (lines 12-17) and reused across subsequent quantization operations to avoid redundant computation.

### The cq_quantize Implementation

The `cq_quantize` function orchestrates the transformation pipeline:

1. **Rotation**: Applies the Hadamard matrix to the weight group
2. **Normalization**: Centers and scales the rotated values
3. **Codebook Mapping**: Quantizes to the nearest Lloyd-Max centroid
4. **Dequantization**: Reconstructs the approximate weight by reversing the rotation

For complete model quantization, `cq_quantize_params` (lines 61-70) walks the JAX pytree, applying this process to every parameter matching the "kernel" or "embedding" patterns while respecting transposed kernel layouts via the `_reduces_second_last` check.

## Activation Quantization and QAT Integration

While weights receive the specialized CQ treatment, activations follow a standard per-element quantization path. The `fake_quant_act` wrapper quantizes activations to **8 bits** (`ACT_BITS = 8`) during both training and inference.

Quantization-aware training (QAT) integrates through the `maybe_quant_weights` and `configure_qat` functions, which inject fake quantization nodes into the forward pass. This allows the model to adapt to quantization noise during fine-tuning, bridging the gap between floating-point training and integer inference.

## Deploying CQ W4A8 Models

### Basic Deployment Workflow

The `deploy_quantize` function in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) (lines 74-81) serves as the primary API for preparing models for inference. When invoked through the CLI export routine in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), it reads the `weight_bits` configuration and returns a quantized parameter set alongside a specification string.

```python
from needle.model.run import load_model
from needle.model.quantize import configure_deploy, deploy_quantize

# Load model and parameters

model, params = load_model("llama-2-7b")

# Configure 8-bit activations with 4-bit weights

configure_deploy(act_bits=8, kv_bits=0)
quantized_params, spec = deploy_quantize(
    params, 
    config=type("Config", (), {"weight_bits": "default=4"})
)

print(f"Quantization spec: {spec}")  # Output: CQ W4

```

### Mixed-Precision Configuration

The `cq_mixed_params` helper enables heterogeneous bit-width assignments across different layers. Using the `parse_bits_map` utility (lines 101-118), you can specify varying precision for attention heads, feed-forward networks, or embedding tables.

```python

# Configure 2-bit attention layers with 4-bit defaults

bits_map = "attn=2,default=4"
quantized_params, spec = deploy_quantize(
    params,
    config=type("Config", (), {"weight_bits": bits_map})
)

```

This flexibility allows aggressive compression of attention weights (which often tolerate lower precision) while preserving higher fidelity for sensitive projection layers.

## Key Implementation Files

The Cactus Quants W4A8 implementation spans several critical modules in the Needle 2 repository:

- **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)** — Contains `cq_quantize`, `cq_quantize_params`, and the Lloyd-Max Gaussian optimizer. This file defines the group-wise rotation logic and codebook caching mechanisms.
- **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)** — Implements the CLI export routine that selects the W4A8 format and coordinates the quantization pipeline before serialization.
- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** — Defines tensor naming conventions used by `canonical_tensor_name` to resolve mixed-precision bit-maps against parameter paths.
- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** — Provides model loading and inference entry points that accept quantized parameter trees.

## Summary

- **CQ W4A8** compresses weights to 4 bits while keeping activations at 8 bits, reducing model size by approximately 50% compared to 8-bit formats.
- **Group-wise Hadamard rotation** (default size 128) decorrelates weights before quantization, preserving accuracy at low bit-widths.
- **Lloyd-Max Gaussian optimization** generates optimal 16-level codebooks for 4-bit weight representation, minimizing reconstruction error.
- **Mixed-precision support** via `parse_bits_map` allows layer-specific quantization strategies (e.g., 2-bit attention with 4-bit MLPs).
- **Integration points** include `deploy_quantize` for inference preparation and `configure_qat` for quantization-aware training.

## Frequently Asked Questions

### What does W4A8 mean in Cactus Quants?

**W4A8** indicates that weights are quantized to **4 bits** while activations remain at **8 bits**. This asymmetric design targets the dominant memory bottleneck in transformer inference—weight loading—while preserving sufficient precision for activation gradients and matrix multiplications on standard hardware.

### How does the Hadamard rotation improve quantization accuracy?

The **Hadamard rotation** applies a normalized Hadamard matrix to weight groups before codebook mapping, distributing magnitude information uniformly across vector elements. This preprocessing reduces the dynamic range within any single dimension, allowing the subsequent Lloyd-Max quantizer to represent the transformed data with lower reconstruction error when compressed to 4-bit centroids.

### Can I use different bit-widths for different layers?

Yes. The `parse_bits_map` function (lines 101-118 in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)) parses configuration strings like `"attn=2,default=4"` to assign specific bit-widths to tensor categories. The `cq_mixed_params` walker then applies appropriate codebooks to attention kernels, feed-forward projections, or embedding tables independently, enabling fine-grained memory-accuracy tradeoffs.

### Where is the quantization code located in the Needle repository?

The primary implementation resides in **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)**, with high-level orchestration in **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**. The quantization utilities depend on [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) for tensor name resolution and are consumed by [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) during inference execution.