# Simple Attention Network Architecture in Needle 2: A Complete Technical Guide

> Explore the Simple Attention Network architecture in Needle 2. Discover its transformer-style encoder, engram memory, and MTP for efficient on-device inference.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-08

---

**The Simple Attention Network (SAN) is the core transformer-style encoder in Needle 2, implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), featuring learned engram memory, multi-token prediction (MTP), and auxiliary contrastive/confidence heads designed for high-performance, on-device inference.**

The Simple Attention Network serves as the primary neural architecture powering Needle 2's language modeling capabilities. This lightweight yet expressive model extends standard transformer encoders with specialized components for long-range retrieval and self-supervised training. According to the cactus-compute/needle source code, the SAN implementation emphasizes efficiency through careful attention mechanisms and optional quantization support.

## Core Architecture Components

The SAN implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) instantiates seven key components during initialization in the `setup()` method.

### Embedding Layer with RMS Scaling

The model begins with a standard token embedding that scales outputs by √*d_model*. As implemented at line 82, `self.embedding` maps input tokens to dense vectors before the scaling operation prepares them for the attention mechanism.

### Multi-Head Attention Stack

The central processing unit is the `Stack` class (line 84), which contains multiple attention blocks. This component handles standard multi-head attention with optional flash-attention optimization and KV-budget management for memory-constrained environments.

### Engram Memory System

A distinctive feature of the SAN architecture is the engram memory system (line 90). These learned tables store compressed n-gram statistics and supply additional key/value pairs to the attention mechanism. This design enables long-range context retrieval without exhausting the KV-cache budget, effectively extending the model's receptive field beyond standard attention windows.

### Multi-Token Prediction Block

The MTP (Multi-Token-Predict) system consists of `self.mtp_combine` (a dense projection layer) and `self.mtp_block` (an additional attention block) initialized at line 96. When activated via `return_mtp=True`, this mechanism concatenates the main hidden state with shifted embeddings, projects them through the dense layer, and processes them through the attention block to improve next-token prediction accuracy.

### Contrastive and Confidence Heads

The architecture includes two specialized heads initialized in `setup()`:

- **Contrastive Head** (line 86): Projects pooled representations to a low-dimensional space for contrastive learning of token-level embeddings
- **Confidence Head** (line 88): Predicts scalar confidence scores for uncertainty estimation and tool-selection logic

### Rotary Positional Embeddings

Rather than learned positional encodings, the SAN uses pre-computed sinusoidal frequencies applied via the `_rope` helper (line 104) to queries and keys, providing positional information without additional parameters.

### Causal and Padding Masks

The forward pass enforces autoregressive behavior through causal masks created by `make_causal_mask` (line 88) and optional padding masks via `make_padding_mask` (line 93), ensuring the model only attends to previous positions and properly handles variable-length sequences.

## Forward Pass Execution Flow

The `__call__` method in `SimpleAttentionNetwork` orchestrates the following computation graph:

1. **Embedding and Scaling**: Input tokens pass through `self.embedding` and multiply by √*d_model*
2. **RoPE Generation**: Cosine/sine tables compute for the current sequence length
3. **Engram KV Retrieval**: Optional extra key/value pairs generated from engram tables at specified layers (2 and 15 by default)
4. **Stack Processing**: The main `self.stack` processes the sequence through attention blocks, returning hidden states
5. **Logits Projection**: Final logits project back onto the token-embedding matrix via weight tying
6. **Optional MTP**: When `return_mtp=True`, the `mtp_block` produces additional prediction logits through the second-stage mechanism

## Source Code Structure

The SAN resides in the `SimpleAttentionNetwork` class defined at line 78 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Key implementation files include:

- [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py): Contains `TransformerConfig`, `SimpleAttentionNetwork`, attention blocks, engram memory, and prediction heads
- [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py): High-level `run` helper for model instantiation and inference execution
- [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py): Fake-quantization implementation for `quant=True` mode
- [`tests/test_inference.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_inference.py): Unit tests validating SAN forward passes and MTP outputs

## Implementation Example

The following examples demonstrate typical usage patterns:

```python

# Model instantiation

from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    contrastive_dim=128,
)

model = SimpleAttentionNetwork(config=cfg)

```

```python

# Standard inference

import jax.numpy as jnp

tokens = jnp.array([[1, 5, 23, 0, 0]])  # batch_size=1, seq_len=5

logits = model(tokens)  # Shape: (1, 5, 8192)

```

```python

# Multi-token prediction mode

logits, mtp_logits = model(tokens, return_mtp=True)

# mtp_logits shape: (1, 5, 8192)

```

```python

# Contrastive encoding

embeddings, log_temp = model.encode_contrastive(tokens)

# embeddings shape: (batch, contrastive_dim)

```

## Summary

- The Simple Attention Network in Needle 2 implements a customized transformer encoder in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) with specialized components for efficient inference
- **Engram memory** provides compressed n-gram statistics as additional attention keys/values, extending context windows without KV-cache exhaustion
- The **MTP block** enables two-stage prediction when `return_mtp=True`, improving next-token accuracy through additional processing
- **Contrastive and confidence heads** support self-supervised training and uncertainty quantification respectively
- **RoPE embeddings** supply positional information without parameter overhead
- The architecture supports optional quantization through [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) for deployment on resource-constrained devices

## Frequently Asked Questions

### What makes the Simple Attention Network different from standard transformer architectures?

The SAN extends conventional transformers with learned engram memory tables that compress n-gram statistics and feed them as additional key/value pairs to the attention mechanism. Unlike standard transformers that rely solely on the KV cache, this approach enables longer effective context windows without proportional memory increases. The architecture also includes a Multi-Token Prediction block for improved next-token accuracy and auxiliary heads for contrastive learning and confidence estimation.

### How does the engram memory system work in practice?

During the forward pass, specifically at layers 2 and 15 by default, the model retrieves compressed representations from learned engram tables. These representations serve as additional keys and values concatenated with the standard attention inputs. Implemented in `self.engrams` at line 90 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this mechanism allows the model to access long-range statistical patterns without storing full history in the KV cache.

### When should I use the Multi-Token Prediction (MTP) feature?

Activate MTP by passing `return_mtp=True` to the model's `__call__` method when you need improved next-token prediction accuracy and can afford the computational cost of the second-stage processing. The MTP block concatenates hidden states with shifted embeddings and runs them through an additional attention block, producing `mtp_logits` alongside standard outputs. This feature is particularly useful during model initialization or when maximum prediction quality is required.

### Where is the Simple Attention Network implemented in the Needle 2 codebase?

The core implementation resides in the `SimpleAttentionNetwork` class at line 78 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This file defines the complete architecture including the `Stack` class for attention blocks, engram memory systems, and auxiliary heads. Supporting functionality appears in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) for high-level inference APIs and [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) for optional quantization support.