How to Export a Needle 2 Model to 4‑Bit: Complete Guide to `.cact` Quantization

To export a Needle 2 model to 4-bit, use the write_export function in needle/model/export.py with bits=4, which applies Cactus-Quants (CQ) packing with Walsh-Hadamard transforms and writes a binary .cact file optimized for on-device inference.

The Needle 2 inference engine from the cactus-compute/needle repository stores weights in a custom Cactus-Quants (CQ) layout that enables efficient matrix multiplication on integer hardware. When you export a trained checkpoint to the 4-bit .cact format, you reduce model size while preserving FP16 precision for layer-normalization, Hadamard diagonals, and gating tensors. This guide covers the complete workflow from checkpoint loading to binary serialization using both CLI and Python APIs.

Understanding the 4-Bit Cactus-Quants Format

The CQ format packs matrix multiplication weights into 4-bit integers while keeping critical activation-related tensors in FP16. According to the source code in needle/model/export.py, the quantization process applies a Walsh-Hadamard transform followed by normalization before encoding indices using nibble-packing (4 bits per weight).

Key characteristics of the format include:

• Group-wise quantization: Tensors are processed in groups of 128 elements by default, configured via the group parameter.
• Mixed precision: Only mat-mul matrices are quantized; layer-norms and embeddings remain in higher precision.
• Zero-copy runtime: The resulting .cact file uses a 120-byte header followed by a tensor directory, allowing the inference engine to memory-map the file with minimal parsing overhead.

Step-by-Step Export Process

The export pipeline in needle/model/export.py follows seven distinct stages when converting a trained checkpoint to 4-bit format.

1. Load the Checkpoint

The checkpoint contains raw TensorFlow/NumPy parameters (params) and a TransformerConfig describing the architecture. Use load_checkpoint from needle/model/run.py to retrieve these components.

2. Resolve Bit-Width Configuration

The exporter defaults to 4-bit quantization (W4A8), but you can override this using the --bits flag or a custom bit-map. Internally, the function parse_bits_map at line 81 of export.py validates and converts the bit specification into per-tensor settings.

3. Quantize Matrices with CQ Packing

Each weight matrix undergoes _cq_pack (implemented around line 84 in export.py), which:

1. Pads the matrix to a multiple of the quantization group size (default 128)
2. Applies the Walsh-Hadamard transform to reduce quantization error
3. Normalizes the transformed values
4. Encodes the indices using 4-bit nibble-packing

4. Build the Tensor Directory

Every tensor is wrapped in a _Tensor object tracking dtype, shape, offset, size, group, and bits. The directory layout is serialized using the _REC_FMT structure defined at lines 98–99 of export.py.

5. Assemble the Binary Header

A fixed 120-byte header encodes the model geometry (vocabulary size, d_model, number of heads) and shared codebooks for CQ groups, as seen in lines 12–30 of export.py.

6. Append Tokenizer Data

If provided, the SentencePiece BPE blob is appended as a RAW tensor using _tokenizer_blob (lines 18–38), ensuring the .cact file remains self-contained.

7. Serialize to Disk

Finally, write_export (lines 87–93) concatenates the header, directory, and tensor blobs into a contiguous binary and writes it to the specified path.

Command-Line Export (CLI)

The recommended approach for manual exports uses the CLI entry point defined in needle/model/export.py at line 36.

python -m needle.model.export \
    --bits 4 \
    checkpoint.pt \
    needle.cact

This command automatically infers the kv_window from the config, validates the tokenizer vocabulary, and applies 4-bit quantization to all compatible weight matrices. The --bits-map flag accepts string specifications like "W4A8" for granular control over weight and activation precision.

Programmatic Export

For pipeline integration or custom training workflows, import the export functions directly.

from needle.model.export import write_export
from needle.model.run import load_checkpoint
from needle.model.tokenizer import get_tokenizer

# Load checkpoint and architecture

params, config, _ = load_checkpoint("checkpoint.pt", return_run=True)

# Match tokenizer to model vocabulary

tokenizer = get_tokenizer(config.vocab_size)

# Export to 4-bit CQ format

info = write_export(
    params,
    config,
    path="needle.cact",
    bits=4,                # Force 4-bit quantization

    group=128,             # Group size for CQ packing

    tokenizer=tokenizer,   # Embed tokenizer blob

)

print(f"Exported {info['path']}: {info['bytes']/1e6:.2f} MB, {info['tensors']} tensors")

Custom Bit-Mappings

For advanced use cases requiring per-tensor precision control, pass a bit-map string to parse_bits_map. This function interprets specifications like "W4A8" (4-bit weights, 8-bit activations) and applies them selectively during the packing phase.

Summary

• Primary export function: write_export in needle/model/export.py handles the complete serialization pipeline.
• Quantization logic: _cq_pack applies Walsh-Hadamard transforms and 4-bit nibble-packing with a default group size of 128.
• CLI entry: python -m needle.model.export --bits 4 provides the fastest path to generate .cact files.
• File structure: The binary format comprises a 120-byte header, a tensor directory using _REC_FMT, and contiguous weight blobs including an optional RAW tokenizer.
• Runtime efficiency: The CQ layout enables single-mega-kernel inference via memory-mapped I/O with zero parsing overhead.

Frequently Asked Questions

What is the difference between CQ and standard INT4 quantization?

Cactus-Quants (CQ) differs from vanilla INT4 by applying a Walsh-Hadamard transform to weights before quantization, which reduces the impact of outliers, and by maintaining FP16 precision for layer-normalization and gating tensors. This hybrid approach preserves model quality better than uniform quantization while achieving the same 4-bit storage density for mat-mul weights.

Can I export Needle 2 models to 8-bit instead of 4-bit?

Yes, change the bits parameter to 8 in the CLI (--bits 8) or Python API (bits=8). The parse_bits_map function supports arbitrary bit widths, though 4-bit (W4A8) is the default and tested configuration for on-device deployment.

Why does the .cact file include the tokenizer?

The exporter optionally embeds the SentencePiece BPE blob as a RAW tensor via _tokenizer_blob, making the .cact file self-contained. This eliminates external file dependencies during inference, allowing the runtime to load both model weights and vocabulary from a single memory-mapped file.

How does the group size affect quantization accuracy?

The default group size of 128 elements balances granularity and overhead. Smaller groups improve quantization precision by allowing finer-scale normalization but increase the storage cost of scale factors. You can adjust this via the group parameter in write_export, though values must divide the tensor dimensions evenly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}