How Needle 2 Model Compresses to a Single Binary: From 45M Parameters to 14MB

Needle 2 compresses its 45-million-parameter Simple Attention Network into a single 14MB binary by combining a compact Hadamard-MLP architecture with custom 2-bit Cactus Quantization and a proprietary .cact container format that bundles quantized weights with a tiny native runtime.

Needle 2 is an edge-optimized language model developed by Cactus Compute that delivers full inference capabilities in a footprint small enough for microcontrollers. According to the cactus-compute/needle source code, the system achieves this through aggressive quantization and architectural trade-offs that preserve model expressiveness while eliminating runtime dependencies.

Stage 1: Compact Architecture Design

The foundation of Needle 2’s compression begins in needle/model/architecture.py, where the Simple Attention Network replaces conventional transformer components with parameter-efficient alternatives.

• Hadamard-MLP replaces standard feed-forward layers, reducing the parameter count while maintaining non-linear expressiveness through Hadamard matrix transformations.
• Grouped-Query Attention (GQA) shares key and value heads across query groups, cutting memory bandwidth and cache size requirements compared to multi-head attention.
• Engram key-value memory adds a compressed external memory module that reduces the need for massive internal weight storage.
• Multi-lane hyper-connections create efficient pathways between layers without the dense parameter growth seen in traditional residual networks.

These architectural choices establish a lean topology that requires fewer floating-point operations before quantization even begins.

Stage 2: CQ-2-Bit Quantization

In needle/model/quantize.py, Needle 2 applies Cactus Quant (CQ), a proprietary 2-bit quantization scheme that learns a small codebook per tensor. Instead of storing full-precision weights, the system stores indices into these learned codebooks.

This approach delivers a 4× reduction compared to 8-bit INT8 quantization and a 16× reduction compared to 32-bit floating point. For the 45-million-parameter model, this compression is the primary driver behind the final 14MB binary size. The quantization algorithm preserves model accuracy by optimizing codebook entries specifically for the Simple Attention Network’s activation patterns.

Stage 3: Baking into the .cact Container

Once quantized, the model moves to needle/model/export.py for containerization. The .cact format is a proprietary binary container that bundles three critical components:

1. Quantized weight blobs – The 2-bit tensor indices packed into dense bitstreams.
2. Static architecture description – A configuration defining the Hadamard-MLP dimensions, GQA groupings, and memory lane structures.
3. Tiny C runtime – A hand-optimized decoder that understands how to reconstruct 2-bit tensors on-the-fly during inference.

Because the runtime is tightly coupled to the Simple Attention layout, it requires only a few megabytes of code, eliminating the need for heavy dependencies like PyTorch or TensorFlow at inference time.

Stage 4: Single-File Delivery

The final artifact is a single file (needle.cact) that serializes architecture, weights, and runtime into one self-contained unit. As implemented in needle/__init__.py, loading this binary creates a ready-to-run engine that occupies ≈14 MB on disk and ≈28 MB of RAM during inference.

Loading the compressed binary:

import needle

# Load the pre-compressed Needle 2 binary (the .cact file)

agent = needle.Needle(weights="needle_v2.cact", tools=[...])

# Run inference – the engine reads the .cact file directly

result = agent.run("What is the weather in Tokyo?")
print(result)

Fine-Tuning Without Decompression

Needle 2 supports parameter-efficient fine-tuning through LoRA adapters that operate directly on the compressed base weights. You can train custom adapters and export them as new single binaries without ever decompressing the original model.

# Fine-tune a LoRA adapter on the compressed base, then re-export

from needle.model.finetune import LoRAAdapter, export_finetuned

adapter = LoRAAdapter(base_weights="needle_v2.cact")
adapter.finetune(dataset=my_data, epochs=3)

# Export the tuned model – still a single .cact binary

export_finetuned(adapter, out_path="my_needle.cact")

Summary

• Compact architecture in needle/model/architecture.py reduces parameter count via Hadamard-MLP, Grouped-Query Attention, and engram memory.
• CQ-2-bit quantization in needle/model/quantize.py achieves 16× compression over FP32 by storing weight indices instead of full values.
• Proprietary containerization in needle/model/export.py bundles quantized weights, static config, and a tiny C runtime into the .cact format.
• Single-binary deployment through needle/__init__.py delivers a 14MB file that runs in 28MB RAM with zero external dependencies.

Frequently Asked Questions

What is the .cact file format in Needle 2?

The .cact format is a proprietary binary container defined in needle/model/export.py that stores the 2-bit quantized weight blobs, static architecture descriptions, and a minimal C runtime decoder. It allows the entire model to exist as a single file that requires no external libraries to execute.

How does CQ-2-bit quantization compare to INT8 or FP16?

CQ-2-bit quantization provides a 4× size reduction compared to 8-bit INT8 and a 16× reduction compared to 32-bit FP32. While INT8 is common for edge deployment, the Cactus Quant scheme in needle/model/quantize.py uses learned codebooks to maintain accuracy at the extreme compression level of 2 bits per weight.

Can I fine-tune Needle 2 without decompressing the binary?

Yes. The needle.model.finetune module supports LoRA (Low-Rank Adaptation) training directly on the compressed .cact weights. You can train adapters and export them as new single binaries using export_finetuned(), preserving the 14MB single-file format.

What hardware can run the Needle 2 single binary?

Any device capable of allocating approximately 28MB of RAM can run the Needle 2 binary. The hand-optimized C runtime has minimal system requirements, making it suitable for microcontrollers, mobile devices, and other constrained edge environments without GPU acceleration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}