# How Needle 2's Simple Attention Network Differs from Traditional Transformers

> Explore how Needle 2's Simple Attention Network differs from transformers, using a Hadamard-based MLP, Grouped-Query Attention, and Engram KV memory for a compact, efficient model.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-29

---

**Needle 2 replaces the standard transformer block with a Simple Attention Network (SAN) that combines a Hadamard-based MLP, Grouped-Query Attention, Engram KV memory, and multi-lane hyper-connections to achieve a 45-million-parameter model running under 30 MB of RAM.**

Needle 2 from Cactus Compute reimagines transformer architecture for extreme edge deployment. Unlike conventional transformers that scale memory linearly with sequence length, the **Simple Attention Network (SAN)** implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) employs weight-free transformations and aggressive memory budgeting to deliver desktop-grade language capabilities on resource-constrained devices.

## HadamardMLP Replaces the Feed-Forward Network

Traditional transformers rely on a **Feed-Forward Network (FFN)** consisting of two `nn.Dense` layers separated by GeLU or ReLU activation. According to the Needle 2 source code, the SAN eliminates this parameter-heavy stack entirely.

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines [87-103](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L87), the `HadamardMLP` class implements a **Walsh-Hadamard transform** followed by learned diagonal scaling matrices. The Hadamard transform operates in `O(n log n)` time without storing weights, dramatically reducing parameter count and memory traffic while maintaining rich feature mixing through the fixed orthonormal `_walsh_matrix` defined at lines [80-85](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L80).

## Grouped-Query Attention Reduces KV Memory

Where standard **Multi-Head Attention (MHA)** maintains separate query, key, and value projections for every head, Needle 2 adopts **Grouped-Query Attention (GQA)** via the `MultiHeadAttention` class at lines [209-277](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L209).

GQA decouples `num_heads` (query heads) from `num_kv_heads` (key/value heads). Fewer KV heads directly reduce the KV cache memory footprint and computational overhead, a critical optimization for staying within the 45-million-parameter budget while preserving expressive power.

## Engram Key-Value Memory for Long-Range Retrieval

Standard transformers store context exclusively in hidden states and limited KV caches. Needle 2 introduces the **Engram** class at lines [81-107](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L81), implementing a learned hash-based KV cache that stores n-gram statistics across the sequence.

This **Engram Key-Value Memory** enables long-range token retrieval without exploding the KV budget, allowing the model to maintain a 256-token sliding window while remembering earlier context through the hashed memory layer.

## Multi-Lane Hyper-Connections

Instead of single-lane residual connections, the SAN splits each transformer layer into `mhc_lanes` (default 4) parallel pathways. The `SimpleAttentionNetwork.setup` method at lines [77-104](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L77) configures these **Multi-Lane Hyper-Connections** via `mhc_*` parameters and `mtp_*` modules within the `Stack` class.

These parallel hyper-connections provide additional information flow pathways without adding trainable parameters, improving capacity-to-size efficiency beyond standard residual streams.

## Z-CRMSNorm and Memory Budgeting

Needle 2 replaces LayerNorm with **Z-CRMSNorm** (zero-centered RMSNorm), implemented at lines [46-55](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L46). RMSNorm omits mean subtraction, making it cheaper to compute and more stable at low-precision (bfloat16) formats essential for the 2-bit quantization pipeline.

To guarantee sub-30 MB runtime memory regardless of conversation length, the SAN implements a **KV-budgeted window** via `kv_budget_window` at lines [998-1012](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L998). This calculation caps the effective KV cache to approximately 11 MiB, ensuring the total RAM footprint remains under 28 MB even as sequences grow.

## Implementation Structure

The SAN architecture spans several core files in the Cactus Compute repository:

- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** – Contains the complete SAN implementation including `HadamardMLP`, `Engram`, `MultiHeadAttention`, `ZCRMSNorm`, and the `SimpleAttentionNetwork` class starting at line [78](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L78).

- **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)** – Implements the 2-bit CQ2 quantization engine that compresses the SAN weights for edge deployment.

- **[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)** – Exposes the public API (`Needle`, `tool`, `extract`) that abstracts the underlying SAN complexity from end users.

## Practical Usage

Despite the architectural innovations, the SAN operates through a simple Python API. The `needle.Needle` class automatically instantiates the underlying Simple Attention Network:

```python
import needle
from pydantic import BaseModel

# Tool-calling with SAN-based inference

@needle.tool
def get_weather(city: str):
    """Return current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What’s the weather like in Lagos?")
print(result["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

# Structured extraction

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)

# → Acme Corp 1200.0

```

These high-level calls trigger the `SimpleAttentionNetwork` forward pass defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), handling the Hadamard transforms, GQA computations, and Engram memory lookups transparently.

## Summary

- **HadamardMLP** replaces dense FFN layers with weight-free Walsh-Hadamard transforms, cutting parameters and memory traffic.
- **Grouped-Query Attention** reduces KV cache size by decoupling query and key/value head counts in `MultiHeadAttention`.
- **Engram memory** provides hash-based long-range retrieval without linearly increasing KV storage.
- **Multi-Lane Hyper-Connections** add parallel residual pathways without extra parameters, configured via `mhc_lanes` in the `Stack` class.
- **Z-CRMSNorm** replaces LayerNorm for stability at 2-bit precision, while `kv_budget_window` caps memory usage at ~11 MiB to maintain sub-30 MB total footprint.

## Frequently Asked Questions

### What makes the HadamardMLP more efficient than a standard transformer FFN?

The **HadamardMLP** eliminates the two large dense weight matrices found in traditional feed-forward networks. By using the fixed **Walsh-Hadamard transform** (`_walsh_matrix` at lines 80-85) followed by diagonal scaling, the operation requires zero stored weights for the mixing component, reducing the 45-million-parameter model's memory footprint while maintaining `O(n log n)` computational complexity.

### How does Needle 2 keep memory usage under 30 MB during inference?

Needle 2 employs three complementary strategies: **Grouped-Query Attention** reduces the number of KV heads, the **KV-budgeted window** (`kv_budget_window` at lines 998-1012) caps the cache at approximately 11 MiB, and **2-bit CQ2 quantization** compresses weights in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). Together, these ensure runtime RAM never exceeds ~28 MB regardless of conversation length.

### Why use Z-CRMSNorm instead of standard LayerNorm?

**Z-CRMSNorm** (zero-centered RMSNorm) omits the mean subtraction step required by LayerNorm, making it computationally cheaper and more numerically stable when operating at low-precision formats like bfloat16. This stability is essential for the 2-bit quantization pipeline used by the Simple Attention Network.

### Can Needle 2's SAN architecture handle long context windows?

Yes, through the **Engram** memory system (lines 81-107), which stores n-gram statistics in a learned hash-based cache. This allows the model to retrieve information from beyond the 256-token sliding window without increasing the KV cache size, effectively extending context range while adhering to strict memory constraints.