# How to Export a Needle 2 Model to 4‑Bit: Complete Guide to `.cact` Quantization

> Export your Needle 2 model to 4-bit using the write_export function with bits=4. Learn how to create optimized .cact files for efficient on-device inference.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-12

---

**To export a Needle 2 model to 4-bit, use the `write_export` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) with `bits=4`, which applies Cactus-Quants (CQ) packing with Walsh-Hadamard transforms and writes a binary `.cact` file optimized for on-device inference.**

The Needle 2 inference engine from the `cactus-compute/needle` repository stores weights in a custom *Cactus-Quants* (CQ) layout that enables efficient matrix multiplication on integer hardware. When you export a trained checkpoint to the 4-bit `.cact` format, you reduce model size while preserving FP16 precision for layer-normalization, Hadamard diagonals, and gating tensors. This guide covers the complete workflow from checkpoint loading to binary serialization using both CLI and Python APIs.

## Understanding the 4-Bit Cactus-Quants Format

The CQ format packs matrix multiplication weights into 4-bit integers while keeping critical activation-related tensors in FP16. According to the source code in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the quantization process applies a **Walsh-Hadamard transform** followed by normalization before encoding indices using nibble-packing (4 bits per weight).

Key characteristics of the format include:

- **Group-wise quantization**: Tensors are processed in groups of 128 elements by default, configured via the `group` parameter.
- **Mixed precision**: Only mat-mul matrices are quantized; layer-norms and embeddings remain in higher precision.
- **Zero-copy runtime**: The resulting `.cact` file uses a 120-byte header followed by a tensor directory, allowing the inference engine to memory-map the file with minimal parsing overhead.

## Step-by-Step Export Process

The export pipeline in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) follows seven distinct stages when converting a trained checkpoint to 4-bit format.

### 1. Load the Checkpoint

The checkpoint contains raw TensorFlow/NumPy parameters (`params`) and a `TransformerConfig` describing the architecture. Use `load_checkpoint` from [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) to retrieve these components.

### 2. Resolve Bit-Width Configuration

The exporter defaults to 4-bit quantization (W4A8), but you can override this using the `--bits` flag or a custom bit-map. Internally, the function `parse_bits_map` at line 81 of [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py) validates and converts the bit specification into per-tensor settings.

### 3. Quantize Matrices with CQ Packing

Each weight matrix undergoes `_cq_pack` (implemented around line 84 in [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py)), which:

1. Pads the matrix to a multiple of the quantization group size (default 128)
2. Applies the Walsh-Hadamard transform to reduce quantization error
3. Normalizes the transformed values
4. Encodes the indices using 4-bit nibble-packing

### 4. Build the Tensor Directory

Every tensor is wrapped in a `_Tensor` object tracking dtype, shape, offset, size, group, and bits. The directory layout is serialized using the `_REC_FMT` structure defined at lines 98–99 of [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py).

### 5. Assemble the Binary Header

A fixed 120-byte header encodes the model geometry (vocabulary size, `d_model`, number of heads) and shared codebooks for CQ groups, as seen in lines 12–30 of [`export.py`](https://github.com/cactus-compute/needle/blob/main/export.py).

### 6. Append Tokenizer Data

If provided, the SentencePiece BPE blob is appended as a RAW tensor using `_tokenizer_blob` (lines 18–38), ensuring the `.cact` file remains self-contained.

### 7. Serialize to Disk

Finally, `write_export` (lines 87–93) concatenates the header, directory, and tensor blobs into a contiguous binary and writes it to the specified path.

## Command-Line Export (CLI)

The recommended approach for manual exports uses the CLI entry point defined in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) at line 36.

```bash
python -m needle.model.export \
    --bits 4 \
    checkpoint.pt \
    needle.cact

```

This command automatically infers the `kv_window` from the config, validates the tokenizer vocabulary, and applies 4-bit quantization to all compatible weight matrices. The `--bits-map` flag accepts string specifications like `"W4A8"` for granular control over weight and activation precision.

## Programmatic Export

For pipeline integration or custom training workflows, import the export functions directly.

```python
from needle.model.export import write_export
from needle.model.run import load_checkpoint
from needle.model.tokenizer import get_tokenizer

# Load checkpoint and architecture

params, config, _ = load_checkpoint("checkpoint.pt", return_run=True)

# Match tokenizer to model vocabulary

tokenizer = get_tokenizer(config.vocab_size)

# Export to 4-bit CQ format

info = write_export(
    params,
    config,
    path="needle.cact",
    bits=4,                # Force 4-bit quantization

    group=128,             # Group size for CQ packing

    tokenizer=tokenizer,   # Embed tokenizer blob

)

print(f"Exported {info['path']}: {info['bytes']/1e6:.2f} MB, {info['tensors']} tensors")

```

### Custom Bit-Mappings

For advanced use cases requiring per-tensor precision control, pass a bit-map string to `parse_bits_map`. This function interprets specifications like `"W4A8"` (4-bit weights, 8-bit activations) and applies them selectively during the packing phase.

## Summary

- **Primary export function**: `write_export` in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) handles the complete serialization pipeline.
- **Quantization logic**: `_cq_pack` applies Walsh-Hadamard transforms and 4-bit nibble-packing with a default group size of 128.
- **CLI entry**: `python -m needle.model.export --bits 4` provides the fastest path to generate `.cact` files.
- **File structure**: The binary format comprises a 120-byte header, a tensor directory using `_REC_FMT`, and contiguous weight blobs including an optional RAW tokenizer.
- **Runtime efficiency**: The CQ layout enables single-mega-kernel inference via memory-mapped I/O with zero parsing overhead.

## Frequently Asked Questions

### What is the difference between CQ and standard INT4 quantization?

Cactus-Quants (CQ) differs from vanilla INT4 by applying a Walsh-Hadamard transform to weights before quantization, which reduces the impact of outliers, and by maintaining FP16 precision for layer-normalization and gating tensors. This hybrid approach preserves model quality better than uniform quantization while achieving the same 4-bit storage density for mat-mul weights.

### Can I export Needle 2 models to 8-bit instead of 4-bit?

Yes, change the `bits` parameter to `8` in the CLI (`--bits 8`) or Python API (`bits=8`). The `parse_bits_map` function supports arbitrary bit widths, though 4-bit (W4A8) is the default and tested configuration for on-device deployment.

### Why does the `.cact` file include the tokenizer?

The exporter optionally embeds the SentencePiece BPE blob as a RAW tensor via `_tokenizer_blob`, making the `.cact` file self-contained. This eliminates external file dependencies during inference, allowing the runtime to load both model weights and vocabulary from a single memory-mapped file.

### How does the group size affect quantization accuracy?

The default group size of 128 elements balances granularity and overhead. Smaller groups improve quantization precision by allowing finer-scale normalization but increase the storage cost of scale factors. You can adjust this via the `group` parameter in `write_export`, though values must divide the tensor dimensions evenly.