Cactus Quants W4A8 Quantization in Needle 2: A Complete Technical Guide

The Cactus Quants (CQ) W4A8 scheme compresses model weights to 4-bit precision while maintaining 8-bit activations through group-wise Hadamard rotation and Lloyd-Max Gaussian codebook optimization.

The Cactus Quants W4A8 quantization format serves as the default deployment configuration in Needle 2, the open-source inference engine developed by Cactus Compute. This hybrid precision approach reduces memory bandwidth pressure by storing weights at 4-bit resolution while preserving full 8-bit arithmetic for activations, striking an optimal balance between compression and hardware efficiency on modern accelerators.

What Is CQ W4A8 Quantization?

CQ W4A8 is a mixed-precision quantization specification where W4 denotes 4-bit weights and A8 denotes 8-bit activations. Unlike uniform quantization that applies the same bit-width across all tensors, the Cactus Quants implementation in needle/model/quantize.py employs sophisticated techniques to minimize precision loss at ultra-low bit-widths.

The scheme targets the memory bottleneck in transformer inference. By compressing weights to 4 bits (achieving approximately 50% size reduction compared to 8-bit formats), the system reduces data movement costs while keeping activations at 8 bits to maintain matmul efficiency on standard GPU tensor cores.

How Weight Quantization Works in Needle 2

The core weight quantization logic resides in the cq_quantize function within needle/model/quantize.py (lines 34-47). This implementation uses a combination of group-wise processing and optimal codebook generation rather than simple linear quantization.

Group-Wise Hadamard Rotation

Before quantization, weight tensors are divided into groups of 128 elements (controlled by CQ_GROUP_SIZE = 128). Each group undergoes a Hadamard rotation using the normalized Hadamard matrix cached in _cq_hadamard_np (lines 20-24).

The rotation spreads information uniformly across the group vector, decorrelating weight magnitudes and improving the signal-to-noise ratio when subsequently mapped to discrete centroids. This preprocessing step is critical for maintaining model accuracy at 4-bit precision.

Lloyd-Max Gaussian Codebook Generation

For 4-bit quantization, Needle generates a codebook of 16 optimal levels (2^4) using the _lloyd_max_gaussian optimizer. This algorithm computes centroid boundaries that minimize mean squared error under a Gaussian distribution assumption, yielding superior reconstruction quality compared to uniform quantization grids.

The resulting codebook is cached in _cq_codebook_np upon first invocation (lines 12-17) and reused across subsequent quantization operations to avoid redundant computation.

The cq_quantize Implementation

The cq_quantize function orchestrates the transformation pipeline:

1. Rotation: Applies the Hadamard matrix to the weight group
2. Normalization: Centers and scales the rotated values
3. Codebook Mapping: Quantizes to the nearest Lloyd-Max centroid
4. Dequantization: Reconstructs the approximate weight by reversing the rotation

For complete model quantization, cq_quantize_params (lines 61-70) walks the JAX pytree, applying this process to every parameter matching the "kernel" or "embedding" patterns while respecting transposed kernel layouts via the _reduces_second_last check.

Activation Quantization and QAT Integration

While weights receive the specialized CQ treatment, activations follow a standard per-element quantization path. The fake_quant_act wrapper quantizes activations to 8 bits (ACT_BITS = 8) during both training and inference.

Quantization-aware training (QAT) integrates through the maybe_quant_weights and configure_qat functions, which inject fake quantization nodes into the forward pass. This allows the model to adapt to quantization noise during fine-tuning, bridging the gap between floating-point training and integer inference.

Deploying CQ W4A8 Models

Basic Deployment Workflow

The deploy_quantize function in needle/model/quantize.py (lines 74-81) serves as the primary API for preparing models for inference. When invoked through the CLI export routine in needle/model/export.py, it reads the weight_bits configuration and returns a quantized parameter set alongside a specification string.

from needle.model.run import load_model
from needle.model.quantize import configure_deploy, deploy_quantize

# Load model and parameters

model, params = load_model("llama-2-7b")

# Configure 8-bit activations with 4-bit weights

configure_deploy(act_bits=8, kv_bits=0)
quantized_params, spec = deploy_quantize(
    params, 
    config=type("Config", (), {"weight_bits": "default=4"})
)

print(f"Quantization spec: {spec}")  # Output: CQ W4

Mixed-Precision Configuration

The cq_mixed_params helper enables heterogeneous bit-width assignments across different layers. Using the parse_bits_map utility (lines 101-118), you can specify varying precision for attention heads, feed-forward networks, or embedding tables.

# Configure 2-bit attention layers with 4-bit defaults

bits_map = "attn=2,default=4"
quantized_params, spec = deploy_quantize(
    params,
    config=type("Config", (), {"weight_bits": bits_map})
)

This flexibility allows aggressive compression of attention weights (which often tolerate lower precision) while preserving higher fidelity for sensitive projection layers.

Key Implementation Files

The Cactus Quants W4A8 implementation spans several critical modules in the Needle 2 repository:

• needle/model/quantize.py — Contains cq_quantize, cq_quantize_params, and the Lloyd-Max Gaussian optimizer. This file defines the group-wise rotation logic and codebook caching mechanisms.
• needle/model/export.py — Implements the CLI export routine that selects the W4A8 format and coordinates the quantization pipeline before serialization.
• needle/model/architecture.py — Defines tensor naming conventions used by canonical_tensor_name to resolve mixed-precision bit-maps against parameter paths.
• needle/model/run.py — Provides model loading and inference entry points that accept quantized parameter trees.

Summary

• CQ W4A8 compresses weights to 4 bits while keeping activations at 8 bits, reducing model size by approximately 50% compared to 8-bit formats.
• Group-wise Hadamard rotation (default size 128) decorrelates weights before quantization, preserving accuracy at low bit-widths.
• Lloyd-Max Gaussian optimization generates optimal 16-level codebooks for 4-bit weight representation, minimizing reconstruction error.
• Mixed-precision support via parse_bits_map allows layer-specific quantization strategies (e.g., 2-bit attention with 4-bit MLPs).
• Integration points include deploy_quantize for inference preparation and configure_qat for quantization-aware training.

Frequently Asked Questions

What does W4A8 mean in Cactus Quants?

W4A8 indicates that weights are quantized to 4 bits while activations remain at 8 bits. This asymmetric design targets the dominant memory bottleneck in transformer inference—weight loading—while preserving sufficient precision for activation gradients and matrix multiplications on standard hardware.

How does the Hadamard rotation improve quantization accuracy?

The Hadamard rotation applies a normalized Hadamard matrix to weight groups before codebook mapping, distributing magnitude information uniformly across vector elements. This preprocessing reduces the dynamic range within any single dimension, allowing the subsequent Lloyd-Max quantizer to represent the transformed data with lower reconstruction error when compressed to 4-bit centroids.

Can I use different bit-widths for different layers?

Yes. The parse_bits_map function (lines 101-118 in needle/model/quantize.py) parses configuration strings like "attn=2,default=4" to assign specific bit-widths to tensor categories. The cq_mixed_params walker then applies appropriate codebooks to attention kernels, feed-forward projections, or embedding tables independently, enabling fine-grained memory-accuracy tradeoffs.

Where is the quantization code located in the Needle repository?

The primary implementation resides in needle/model/quantize.py, with high-level orchestration in needle/model/export.py. The quantization utilities depend on needle/model/architecture.py for tensor name resolution and are consumed by needle/model/run.py during inference execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}