How Needle 2's Simple Attention Network Differs from Traditional Transformers

Needle 2 replaces the standard transformer block with a Simple Attention Network (SAN) that combines a Hadamard-based MLP, Grouped-Query Attention, Engram KV memory, and multi-lane hyper-connections to achieve a 45-million-parameter model running under 30 MB of RAM.

Needle 2 from Cactus Compute reimagines transformer architecture for extreme edge deployment. Unlike conventional transformers that scale memory linearly with sequence length, the Simple Attention Network (SAN) implemented in needle/model/architecture.py employs weight-free transformations and aggressive memory budgeting to deliver desktop-grade language capabilities on resource-constrained devices.

HadamardMLP Replaces the Feed-Forward Network

Traditional transformers rely on a Feed-Forward Network (FFN) consisting of two nn.Dense layers separated by GeLU or ReLU activation. According to the Needle 2 source code, the SAN eliminates this parameter-heavy stack entirely.

In needle/model/architecture.py lines 87-103, the HadamardMLP class implements a Walsh-Hadamard transform followed by learned diagonal scaling matrices. The Hadamard transform operates in O(n log n) time without storing weights, dramatically reducing parameter count and memory traffic while maintaining rich feature mixing through the fixed orthonormal _walsh_matrix defined at lines 80-85.

Grouped-Query Attention Reduces KV Memory

Where standard Multi-Head Attention (MHA) maintains separate query, key, and value projections for every head, Needle 2 adopts Grouped-Query Attention (GQA) via the MultiHeadAttention class at lines 209-277.

GQA decouples num_heads (query heads) from num_kv_heads (key/value heads). Fewer KV heads directly reduce the KV cache memory footprint and computational overhead, a critical optimization for staying within the 45-million-parameter budget while preserving expressive power.

Engram Key-Value Memory for Long-Range Retrieval

Standard transformers store context exclusively in hidden states and limited KV caches. Needle 2 introduces the Engram class at lines 81-107, implementing a learned hash-based KV cache that stores n-gram statistics across the sequence.

This Engram Key-Value Memory enables long-range token retrieval without exploding the KV budget, allowing the model to maintain a 256-token sliding window while remembering earlier context through the hashed memory layer.

Multi-Lane Hyper-Connections

Instead of single-lane residual connections, the SAN splits each transformer layer into mhc_lanes (default 4) parallel pathways. The SimpleAttentionNetwork.setup method at lines 77-104 configures these Multi-Lane Hyper-Connections via mhc_* parameters and mtp_* modules within the Stack class.

These parallel hyper-connections provide additional information flow pathways without adding trainable parameters, improving capacity-to-size efficiency beyond standard residual streams.

Z-CRMSNorm and Memory Budgeting

Needle 2 replaces LayerNorm with Z-CRMSNorm (zero-centered RMSNorm), implemented at lines 46-55. RMSNorm omits mean subtraction, making it cheaper to compute and more stable at low-precision (bfloat16) formats essential for the 2-bit quantization pipeline.

To guarantee sub-30 MB runtime memory regardless of conversation length, the SAN implements a KV-budgeted window via kv_budget_window at lines 998-1012. This calculation caps the effective KV cache to approximately 11 MiB, ensuring the total RAM footprint remains under 28 MB even as sequences grow.

Implementation Structure

The SAN architecture spans several core files in the Cactus Compute repository:

• needle/model/architecture.py – Contains the complete SAN implementation including HadamardMLP, Engram, MultiHeadAttention, ZCRMSNorm, and the SimpleAttentionNetwork class starting at line 78.

• needle/model/quantize.py – Implements the 2-bit CQ2 quantization engine that compresses the SAN weights for edge deployment.

• needle/__init__.py – Exposes the public API (Needle, tool, extract) that abstracts the underlying SAN complexity from end users.

Practical Usage

Despite the architectural innovations, the SAN operates through a simple Python API. The needle.Needle class automatically instantiates the underlying Simple Attention Network:

import needle
from pydantic import BaseModel

# Tool-calling with SAN-based inference

@needle.tool
def get_weather(city: str):
    """Return current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What’s the weather like in Lagos?")
print(result["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

# Structured extraction

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)

# → Acme Corp 1200.0

These high-level calls trigger the SimpleAttentionNetwork forward pass defined in needle/model/architecture.py, handling the Hadamard transforms, GQA computations, and Engram memory lookups transparently.

Summary

• HadamardMLP replaces dense FFN layers with weight-free Walsh-Hadamard transforms, cutting parameters and memory traffic.
• Grouped-Query Attention reduces KV cache size by decoupling query and key/value head counts in MultiHeadAttention.
• Engram memory provides hash-based long-range retrieval without linearly increasing KV storage.
• Multi-Lane Hyper-Connections add parallel residual pathways without extra parameters, configured via mhc_lanes in the Stack class.
• Z-CRMSNorm replaces LayerNorm for stability at 2-bit precision, while kv_budget_window caps memory usage at ~11 MiB to maintain sub-30 MB total footprint.

Frequently Asked Questions

What makes the HadamardMLP more efficient than a standard transformer FFN?

The HadamardMLP eliminates the two large dense weight matrices found in traditional feed-forward networks. By using the fixed Walsh-Hadamard transform (_walsh_matrix at lines 80-85) followed by diagonal scaling, the operation requires zero stored weights for the mixing component, reducing the 45-million-parameter model's memory footprint while maintaining O(n log n) computational complexity.

How does Needle 2 keep memory usage under 30 MB during inference?

Needle 2 employs three complementary strategies: Grouped-Query Attention reduces the number of KV heads, the KV-budgeted window (kv_budget_window at lines 998-1012) caps the cache at approximately 11 MiB, and 2-bit CQ2 quantization compresses weights in needle/model/quantize.py. Together, these ensure runtime RAM never exceeds ~28 MB regardless of conversation length.

Why use Z-CRMSNorm instead of standard LayerNorm?

Z-CRMSNorm (zero-centered RMSNorm) omits the mean subtraction step required by LayerNorm, making it computationally cheaper and more numerically stable when operating at low-precision formats like bfloat16. This stability is essential for the 2-bit quantization pipeline used by the Simple Attention Network.

Can Needle 2's SAN architecture handle long context windows?

Yes, through the Engram memory system (lines 81-107), which stores n-gram statistics in a learned hash-based cache. This allows the model to retrieve information from beyond the 256-token sliding window without increasing the KV cache size, effectively extending context range while adhering to strict memory constraints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}