How Tensors Are Organized in the Needle .cact File Format

The .cact file format uses a three-part binary structure comprising a fixed-size header, a tensor directory with per-tensor metadata records, and aligned blob data containing IEEE-754 FP16/FP32 values, CQ-quantized packed weights, or RAW byte sequences.

The .cact archive is the native serialization format for the cactus-compute/needle inference engine, designed for efficient storage and random-access loading of transformer model weights. All tensor organization logic is implemented in needle/model/export.py, which defines deterministic packing routines for quantized and full-precision parameters alongside embedded tokenizer vocabularies.

The Archive Layout Structure

A .cact file follows a strict sequential layout that enables zero-copy slicing during model initialization:

| Header | Tensor Directory | Aligned Tensor Blobs |

This organization eliminates the need for external indexing structures while supporting mixed-precision storage and custom quantization schemes.

Binary Header Specification

The archive begins with a fixed-size header block defined by the _HDR_FMT struct in needle/model/export.py. According to the _pack_cact function (lines 540‑560), this header encodes:

• File tag identifying the format version
• Model configuration: vocabulary size, hidden dimension, attention heads, and maximum sequence length
• Quantization metadata: code-book size for CQ tensors, plus kv_window and kv_bits parameters for key-value cache optimization

The header size remains constant regardless of model scale, allowing the loader to parse critical configuration values without scanning the entire file.

Tensor Directory Records

Immediately following the header, the tensor directory contains one fixed-length record per tensor. Each record follows the _REC_FMT struct layout and is assembled during the directory construction phase in _pack_cact (lines 667‑72). The _Tensor class (lines 219‑226) serves as the internal container for these entries.

Each directory record stores the following metadata fields:

• dtype: Data type identifier (FP16 = 0, FP32 = 1, CQ = 2, RAW = 3)
• ndim: Number of tensor dimensions (up to 4)
• shape: Dimension sizes (zero-filled for unused axes)
• offset: Byte offset from file start to the tensor's blob
• nbytes: Length of the binary blob in bytes
• group: Group size for CQ-quantized tensors (0 otherwise)
• bits: Bit-width for quantization (0 for unquantized tensors)

The directory is built by iterating over the list returned from the _tensors helper function, which generates a _Tensor instance for every weight matrix, bias vector, and the special tokenizer blob.

Tensor Blob Storage and Alignment

Following the directory, the actual tensor data is written sequentially with strict memory alignment. The _pack_cact function writes blobs in the final loop (lines 74‑78), applying a 64-byte alignment constraint via the _align helper to optimize SIMD access patterns.

The binary format varies by data type:

• FP16/FP32: Raw IEEE-754 floating-point representation
• CQ (Compressed Quantized): Packed integer weights followed by FP16 normalization matrices (packed.tobytes() + norms.tobytes())
• RAW: Unmodified byte sequences (used exclusively for the tokenizer)

Each blob is placed at the byte offset specified in its directory record, enabling the loader to memory-map individual tensors without parsing intermediate structures.

Tokenizer Encoding

The tokenizer is embedded as a special RAW tensor named "tokenizer" (added in _pack_cact at lines 443‑46). Unlike weight matrices, this blob uses custom header structures (_TK_HDR and _TK_REC) to store:

• Total piece count in the vocabulary
• Special token IDs (control, unknown, byte markers)
• Per-piece metadata including UTF-8 byte sequences and type tags (control, unknown, user-defined, or normal)

This design ensures the model archive remains self-contained, eliminating external file dependencies during deployment.

Loading and Parsing .cact Files

The read_export function (lines 96‑104) implements the inverse operation, reconstructing the model state from disk:

1. Parses the header to extract configuration and quantization parameters
2. Rebuilds the code-book for CQ tensors
3. Walks the tensor directory, slicing blobs according to recorded offset and nbytes values
4. Converts binary data back to NumPy arrays (or raw bytes for the tokenizer)

The function returns a tuple of (metadata, tensors), where tensors is a list of NumPy arrays corresponding to the directory order.

Working with .cact Files in Python

Export a checkpoint to the .cact format with quantization:

from needle.model.export import write_export

# Parameters and config loaded from a checkpoint (e.g., .pkl file)

write_export(params, config, path="model.cact", bits=4, group=128)

Load tensors from an existing archive:

from needle.model.export import read_export

metadata, tensors = read_export("model.cact")
print(f"Loaded {len(tensors)} tensors from {metadata['vocab_size']} vocab model")

# Access the embedding matrix (typically the first tensor)

embeddings = tensors[0]  # Shape: (vocab_size, d_model)

print(f"Embedding shape: {embeddings.shape}")

Decode the embedded tokenizer without external files:

from needle.model.export import RefTokenizer

tok = RefTokenizer.from_cact("model.cact")
print(f"Vocabulary contains {len(tok.pieces)} pieces")

Summary

• The .cact format organizes tensors into three contiguous regions: a fixed binary header, a metadata directory, and aligned data blobs.
• Each tensor record specifies data type, shape, quantization parameters, and file offset, supporting FP16, FP32, CQ-quantized, and RAW storage modes.
• The 64-byte alignment of tensor blobs ensures efficient memory mapping and SIMD compatibility during inference.
• Tokenizers are embedded as special RAW tensors with custom header structures, making archives fully self-contained.
• All serialization logic resides in needle/model/export.py, with write_export and read_export providing the primary public API.

Frequently Asked Questions

What is the maximum tensor dimensionality supported in .cact files?

The format supports up to four dimensions per tensor. The shape field in each directory record allocates four slots, with unused dimensions zero-filled. This accommodates standard transformer weight matrices while maintaining fixed-size directory entries.

How does the format store quantized model weights?

CQ (Compressed Quantized) tensors store packed integer weights followed by FP16 normalization matrices. The directory entry records the group size and bits width, allowing the loader to reconstruct floating-point values using the code-book defined in the header.

Where is the tokenizer vocabulary stored within the archive?

The tokenizer exists as a special RAW tensor named "tokenizer" with its own internal header structure (_TK_HDR/_TK_REC). It encodes vocabulary pieces as UTF-8 byte sequences with type tags, eliminating external vocabulary file dependencies.

Why are tensor blobs aligned to 64-byte boundaries?

The _align function enforces 64-byte alignment to optimize cache line utilization and enable SIMD instruction sets during inference. This alignment occurs during export in _pack_cact and is respected during loading to prevent misaligned memory access.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}