What Is the .cact Format and How Does the Export Process Work in Needle?

The .cact file is Needle’s compact binary representation of a trained transformer model, packing all weights, Cactus-Quant metadata, and the tokenizer into a single memory-mapped blob that the C++ inference engine loads without parsing.

Needle (from the cactus-compute/needle repository) stores trained models in the .cact format to enable instant loading and inference. Unlike JSON or pickle-based checkpoints, this binary archive embeds quantization codebooks, tensor geometry, and the tokenizer into a deterministic layout that the runtime can mmap directly.

Architecture of the .cact Format

The .cact archive is structured as a contiguous byte stream with seven distinct regions. According to the source code in needle/model/export.py, the layout enables single-pass streaming without index files.

Header (120 bytes)

The first 120 bytes encode the model’s architecture and quantization settings. In export.py lines 19‑31, the header consists of 30 uint32 fields plus one float32 (rope_theta) that describe vocabulary size, hidden dimensions, number of attention heads, and the codebook length.

Codebooks

Following the header, the file stores concatenated Lloyd-Max codebooks used for Cactus-Quant (CQ) compression. The header field codebook_len specifies the total size of this region. These codebooks support INT4, INT3, and INT2 quantization as defined by CB_BITS = (2, 3, 4) in lines 25‑30 of export.py.

Tensor Directory

A nameless directory of REC_SIZE‑byte records begins after the codebooks. Each record, detailed in lines 33‑38 of export.py, stores a tensor’s dtype, number of dimensions (ndim), shape array, byte offset, size in bytes, group size, and bit-width. This lets the runtime locate any tensor via fixed-size arithmetic without string lookups.

Tensor Order

Tensors appear in a hard-coded canonical sequence so the engine can stream them sequentially. As implemented in lines 38‑44 of export.py, the order is:

• Embedding weights
• Per-layer tensors (input norms, Q/K/V projections, output projections, post-attention norms)
• Multi-head-cache blocks
• Engram tables
• Final normalization weights
• Optional probe heads
• Raw tokenizer blob

Probe Heads

If the checkpoint contains LoRA-style adapters or auxiliary classifiers, _head_tensors (lines 44‑50) appends contrastive and confidence heads after final_norm. The directory records how many probe heads exist, followed by their projection and bias tensors.

Quantization Layout

CQ-quantized tensors store packed indices LSB-first, accompanied by per-group L2 norms. The format supports ternary (5-bit record) fallback and the three standard bit-widths. Lines 51‑65 of export.py handle the bit-packing logic that converts FP16 weight matrices into these compressed representations.

Tokenizer Blob

At the very end of the file, a raw RAW tensor contains a self-contained SentencePiece tokenizer. This blob stores piece counts, scores, types, and UTF-8 strings. The runtime reconstructs a RefTokenizer from this section without external vocabulary files, as shown in lines 68‑74 of export.py.

The Needle Export Process

The export pipeline transforms a PyTorch or Flax checkpoint into a .cact archive through eight distinct stages. The entry point write_export in needle/model/export.py orchestrates this conversion.

1. Load checkpoint – The main() function calls load_checkpoint from needle/model/run.py (lines 37‑43) to deserialize weights into NumPy arrays and a TransformerConfig object.

2. Determine geometry – _geometry validates head dimensions, engram settings, and KV-cache parameters, raising exceptions for unsupported configurations (lines 102‑119).

3. Build tensor list – _tensors creates a mixed-precision list: FP16 tensors (_fp16) for biases and layer norms, and CQ tensors (_q) for weight matrices. It packs each matrix using the specified bit-width and group size (lines 40‑64).

4. Add optional heads – _head_tensors (lines 92‑107) appends probe-head tensors if the model includes contrastive or confidence adapters.

5. Pack tokenizer – _tokenizer_blob (lines 17‑35) serializes the SentencePiece tokenizer into the raw blob format required by the runtime.

6. Assemble header – _pack_cact constructs the 120-byte header, appends the concatenated codebooks, and calculates each tensor’s byte offset aligned to 64-byte boundaries (lines 53‑61).

7. Write directory and data – The export logic writes the REC_SIZE directory records after the header, followed by zero-padding to each tensor’s offset and then the actual tensor blobs (lines 67‑78).

8. Emit file – write_export flushes the final bytes object to disk (default needle.cact) and returns file metadata (lines 86‑93).

Exporting via CLI

You can generate a .cact file directly from the command line:

# Export a checkpoint to a tuned .cact archive

needle build checkpoints/needle2.pkl --out tuned.cact

Programmatic Export

For custom pipelines, import the exporter directly:

from needle.model.export import write_export
from needle.model.run import load_checkpoint
from needle.model.quantize import parse_bits_map

# 1. Load checkpoint into NumPy arrays

params, config, _ = load_checkpoint("checkpoints/needle2.pkl", return_run=True)

# 2. Configure quantization (e.g., 4-bit INT4)

bits = parse_bits_map("4")

# 3. Write the archive

info = write_export(
    params=params,
    config=config,
    path="my_model.cact",
    bits=bits,
    group=128,
    tokenizer=None,  # loads default automatically

)

print(f"Wrote {info['path']} ({info['bytes']/1e6:.2f} MB, {info['tensors']} tensors)")

Reading a .cact File

To inspect or validate an archive:

from needle.model.export import read_export

metadata, tensors = read_export("my_model.cact")
print(metadata)      # Architecture, kv_window, quantization settings

print(len(tensors))  # Total tensor count including tokenizer

Summary

• The .cact format is a self-contained binary archive for Needle transformer models, embedding weights, quantization codebooks, and tokenizers.
• The 120-byte header in needle/model/export.py encodes all architecture geometry and quantization parameters.
• Cactus-Quant (CQ) compression supports INT4, INT3, and INT2 via Lloyd-Max codebooks stored contiguously after the header.
• The tensor directory uses fixed-size records (REC_SIZE) to enable O(1) tensor lookup without name hashing.
• The export process (write_export) loads checkpoints via needle/model/run.py, validates geometry, packs tensors, and aligns all data to 64-byte boundaries for efficient memory mapping.

Frequently Asked Questions

What quantization bit-widths does the .cact format support?

The format supports 2-bit, 3-bit, and 4-bit integer quantization through Cactus-Quant (CQ), with configurable group sizes for per-block scaling. It also includes a 5-bit ternary fallback for special cases, as defined by CB_BITS in needle/model/export.py.

How does the Needle runtime locate specific tensors without names?

The runtime uses the tensor directory stored immediately after the header. Each tensor occupies a fixed-size record (REC_SIZE bytes) containing its byte offset, dimensions, and data type. Because tensors follow a hard-coded canonical order (embedding → layers → cache → tokenizer), the engine can compute addresses directly without parsing names.

Can I export a model programmatically without using the CLI?

Yes. Import write_export from needle.model.export and load_checkpoint from needle.model.run. Pass the loaded parameters and configuration to write_export along with your desired bit-width and group size. This approach is essential for integrating Needle exports into custom training pipelines.

Why does the .cact file include the tokenizer as a raw blob?

Embedding the tokenizer eliminates external file dependencies. The final section of the archive stores a serialized SentencePiece vocabulary (piece count, scores, types, and UTF-8 strings). This allows the C++ inference engine to reconstruct a RefTokenizer directly from the mmap’d file without reading secondary assets from disk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}