Needle 2 45M Parameter Model: Comparing Compact Foundation Models

Needle 2 delivers competitive tool-calling and structured extraction capabilities using only 45 million 4-bit quantized parameters, making it 5–6× smaller than comparable foundation models like FunctionGemma 270M while fitting entirely within a 14MB binary footprint.

The cactus-compute/needle repository positions Needle 2 as a highly efficient alternative to resource-intensive foundation models. By leveraging a custom 4-bit quantization format referred to as "4SM" in the project documentation, this 45M-parameter engine trades wins with models nearly six times its size. According to the source code in README.md, Needle 2 achieves functional parity with competitors using just 2 bits against their f16 precision, enabling deployment on memory-constrained devices.

Understanding the 45M Parameter Architecture

Needle 2 implements a Simple Attention Network architecture optimized for minimal footprint without sacrificing core capabilities. The model's compression strategy centers on aggressive quantization techniques that preserve functional accuracy.

The 4SM Quantization Format

In needle/model/architecture.py, the codebase defines the quantization pipeline responsible for compressing the full-precision model down to 4 bits. This "4SM" format (referenced in the project paper) reduces the model footprint to approximately 14MB for the complete engine binary. The architecture file implements custom kernels that maintain inference speed despite the reduced precision, allowing the 45 million parameters to execute efficiently on standard hardware.

Parameter Count Comparison Against Foundation Models

The repository's README.md explicitly benchmarks Needle 2 against larger foundation models, demonstrating that parameter count does not directly correlate with task performance:

• FunctionGemma 270M: At 270 million parameters, this model is approximately 6× larger than Needle 2, yet the README notes that Needle 2 "trades wins" with FunctionGemma on tool-calling tasks.
• LFM 2.5 230M: With 230 million parameters, this competitor is roughly 5× larger than Needle 2's 45M count.
• Apple FM: Falling within the 200M–250M parameter range, Apple's foundation model is 5–6× larger while offering comparable functionality.

These comparisons, documented in lines 5–6 of README.md, establish Needle 2 as operating within a niche of 5× to 70× smaller models relative to mainstream alternatives, depending on specific configuration and precision requirements.

Running Inference with the Compact Model

Despite its reduced size, Needle 2 exposes a full-featured API through needle/__init__.py for tool-calling and structured data extraction.

Loading the 45M Parameter Engine

The following example demonstrates initializing the 4-bit model and executing a multi-tool query:

import needle

# Load the pre‑downloaded 4‑bit engine (14 MB) and a set of tools

agent = needle.Needle(
    weights="needle2.cact",          # 45 M‑parameter, 4‑bit model

    tools=[get_weather, set_lights] # user‑defined tool functions

)

# Simple tool‑calling query

result = agent.run("What’s the weather in Nairobi and dim the living‑room lights to 20 %")
print(result["results"])

# → [{'city': 'Nairobi', 'temp_c': 23, 'sky': 'cloudy'},

#    {'room': 'living_room', 'brightness': 20}]

Structured Extraction

For data parsing tasks, the model supports Pydantic schema validation as implemented in the package entry point:

# Structured extraction using a Pydantic schema

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

extracted = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-10-15", Invoice
)
print(extracted.vendor, extracted.total)

# → Acme Corp 1200.0

Performance Characteristics at 45M Parameters

Needle 2 challenges the scaling laws that dominate modern foundation model development. While competitors like FunctionGemma 270M and LFM 2.5 230M utilize f16 (16-bit) floating-point precision, Needle 2 operates at 2-bit effective compression in its 4SM format. This architectural choice enables the model to load entirely into L1/L2 cache on modern processors, reducing memory bandwidth bottlenecks that typically constrain larger models.

According to the documentation in doc/apis.md, the reduced parameter count does not limit the model's ability to handle complex tool definitions or multi-step reasoning chains. The 14MB binary size allows for instant startup times and embedded deployment scenarios impossible with 200M+ parameter alternatives.

Summary

Needle 2 demonstrates that foundation model effectiveness depends on architectural efficiency rather than raw parameter volume:

• 45 million parameters with 4-bit quantization deliver comparable performance to 200M+ parameter models
• 14MB binary footprint enables edge deployment and cache-resident inference
• 5–6× smaller than FunctionGemma 270M and LFM 2.5 230M while maintaining functional parity
• Simple Attention Network architecture defined in needle/model/architecture.py optimizes for memory-constrained environments

Frequently Asked Questions

How does Needle 2 achieve competitive performance with only 45M parameters?

Needle 2 utilizes a Simple Attention Network architecture combined with aggressive 4SM quantization (4-bit) that reduces memory bandwidth requirements. By implementing custom inference kernels in needle/model/architecture.py, the model maximizes the efficiency of each parameter, allowing 45M 4-bit weights to perform comparably to 200M+ 16-bit models on tool-calling tasks.

What is the 4SM quantization format mentioned in the Needle 2 paper?

The 4SM format is a proprietary 4-bit quantization scheme referenced in the Needle 2 research documentation and implemented in needle/model/architecture.py. It compresses the model to approximately 14MB (compared to 400MB+ for f16 alternatives) while preserving enough precision for accurate function calling and structured extraction.

Can Needle 2 run on edge devices given its small footprint?

Yes, the 14MB binary size and 45M parameter count make Needle 2 ideal for edge deployment. The entire model fits within standard CPU cache levels, eliminating the GPU requirements and large RAM allocations necessary for 200M+ parameter foundation models like Apple FM or FunctionGemma.

How do I load the 45M parameter model in my application?

Import the Needle class from needle/__init__.py and specify the 4-bit weights file: agent = needle.Needle(weights="needle2.cact", tools=[your_function]). This loads the 45M-parameter engine with the 4SM quantization format ready for inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}