How Small Is the Needle 2 Model? Parameters and Memory Footprint Explained

Needle 2 contains approximately 45 million parameters, ships as a single 14 MB binary, and requires roughly 28 MB of RAM during full inference sessions regardless of conversation length.

The cactus-compute/needle repository provides a deliberately tiny foundation model optimized for on-device tool-calling and structured extraction. Understanding Needle 2 model size is critical for deploying LLM capabilities to resource-constrained edge devices and embedded systems.

Needle 2 Model Specifications

Parameter Count and Binary Size

According to the project's README, Needle 2 is built with approximately 45 million parameters. Despite this capacity, the entire model is packaged as a single 14 MB binary file. This extreme compression is achieved through the CQ2-bit format via Cactus Quants, which stores weights in a highly compressed quantization scheme.

Memory Footprint During Inference

When running a full inference session, the engine allocates roughly 28 MB of RAM. This consumption remains constant regardless of conversation length due to the model's sliding-window memory design. The 256-token sliding window ensures that runtime memory does not scale with context length, making Needle 2 predictable for tiny device deployments.

Architectural Innovations Behind the Tiny Footprint

Simple Attention Network Architecture

In needle/model/architecture.py, the core implementation reveals a Simple Attention Network that replaces the traditional feed-forward network with a Hadamard-based MLP. This architectural choice reduces parameter count while maintaining representational capacity. The implementation also utilizes Grouped-Query Attention (GQA), which shares key and value heads across query groups to further reduce memory bandwidth and cache footprint.

Compressed Quantization with Cactus Quants

The CQ2-bit quantization format mentioned in the source documentation compresses the 45 million parameters into the 14 MB binary. This custom quantization scheme is specifically designed for the Needle runtime, allowing dequantization to happen efficiently during the forward pass without exploding the memory footprint.

Running Inference Within 28 MB RAM

The public API in needle/__init__.py provides straightforward access to load and run the model. The following examples demonstrate how the 28 MB footprint enables practical inference on constrained hardware:

Load the base Needle 2 model and run simple completion:

import needle

# Load the base Needle 2 model (weights are fetched automatically)

agent = needle.Needle(tools=[])

# Simple inference – the model will run within ~28 MB RAM

response = agent.run("What is the capital of France?")
print(response["generated_text"])

Perform structured extraction using the same tiny engine:

# Using the extraction API – also runs on the tiny engine

from pydantic import BaseModel

class Weather(BaseModel):
    city: str
    temperature_c: float
    sky: str

result = needle.extract(
    "Today in Berlin it's 22°C and sunny.", Weather
)
print(result.city, result.temperature_c, result.sky)

Fine-tune while maintaining the original footprint:

# Fine-tuning a model while keeping the same footprint

# (requires the LoRA extras, but the final .cact remains a single file)

needle finetune data.jsonl --epochs 5   # produces a tuned .cact ~14 MB

The KV cache management in needle/model/architecture.py (lines 13-14) pins tool schemas as KV sinks and limits active context to the 256-token sliding window. This design ensures that the 28 MB RAM requirement remains constant even as conversation state grows.

Summary

• Needle 2 contains 45 million parameters according to the repository README.
• The model ships as a single 14 MB binary using CQ2-bit quantization via Cactus Quants.
• Full inference requires approximately 28 MB of RAM, remaining constant regardless of conversation length.
• The Simple Attention Network architecture with Hadamard-based MLP and Grouped-Query Attention enables this efficiency.
• A 256-token sliding window in needle/model/architecture.py prevents memory scaling with context length.

Frequently Asked Questions

How many parameters does Needle 2 have?

Needle 2 contains approximately 45 million parameters. This is explicitly stated in the repository's README at lines 3-6, positioning it as a tiny foundation model suitable for edge deployment while maintaining enough capacity for tool-calling tasks.

What is the file size of the Needle 2 binary?

The entire Needle 2 model is packaged as a single 14 MB binary (.cact file). This extreme compression is achieved through the CQ2-bit quantization format, allowing the 45 million parameters to fit into a file small enough for bandwidth-constrained deployments.

How much RAM does Needle 2 require for inference?

Needle 2 requires roughly 28 MB of RAM during full inference sessions. This footprint remains constant regardless of conversation length due to the sliding-window memory design implemented in needle/model/architecture.py.

What architecture enables Needle 2's small memory footprint?

The Simple Attention Network architecture replaces traditional feed-forward layers with a Hadamard-based MLP and utilizes Grouped-Query Attention (GQA). Additionally, the implementation uses a 256-token sliding window that prevents KV cache expansion, ensuring the 28 MB RAM requirement stays fixed as described in the architecture source file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}