# How Needle 2's Simple Attention Network Architecture Works: A Complete Technical Guide

> Explore Needle 2's Simple Attention Network architecture. Learn how JAX/Flax transformer combines self-attention, Engram memory, Hadamard-MLP, and MTP heads for efficient language modeling.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-02

---

**Needle 2's Simple Attention Network (SAN) is a JAX/Flax-based transformer that combines standard multi-head self-attention with novel Engram memory caching, Hadamard-MLP layers, and auxiliary Multi-Token Prediction heads to achieve efficient, scalable language modeling.**

The Needle 2 repository implements the Simple Attention Network as its core language model in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This Needle 2 Simple Attention Network architecture extends the classic transformer stack with specialized optimizations for memory efficiency and multi-task learning while maintaining compatibility with standard attention mechanisms. Below is a comprehensive walkthrough of the SAN pipeline, from token embedding to final logits.

## Configuration and Initialization

All hyperparameters governing the Simple Attention Network are centralized in the `TransformerConfig` dataclass defined at lines 58‑83 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This configuration object specifies `vocab_size`, `d_model`, `num_heads`, `num_kv_heads`, `num_layers`, and specialized fields like `rope_theta` and `engram_*` parameters that control optional memory augmentation.

During initialization, `SimpleAttentionNetwork` uses this config to instantiate the embedding layer, pre-compute rotary positional encodings, and construct the transformer stack with optional Engram modules. The embedding layer appears in the `setup` method at lines 78‑85, utilizing `nn.Embed` with vectors scaled by √`d_model` for numerical stability.

## Core Architecture Components

### Rotary Positional Embeddings (RoPE)

The SAN implements sinusoidal rotary positional encoding via the `_rope` helper function (`precompute_rope_freqs`) at lines 104‑108. These frequencies are pre-computed for each head dimension and applied to queries and keys within the attention blocks, providing relative positional information without additional learned parameters.

### The Stack and Block Structure

The transformer core utilizes a `Stack` module defined at lines 76‑86 that repeatedly applies `Block` layers using `nn.scan` for memory-efficient iteration across `num_layers`. Each `Block` (lines 6‑34) executes the following operations:

- **Optional Engram injection** that augments hidden states with retrieved memory values
- **ZCRMSNorm** (a specialized LayerNorm variant) followed by **MultiHeadAttention**
- Residual connection with a learned gating scalar
- **Hadamard-MLP**, an efficient feed-forward layer utilizing orthogonal matrix transformations

### Multi-Head Attention Mechanism

The `MultiHeadAttention` class at lines 9‑78 handles query, key, and value projections with support for optional Flash Attention. After computing attention weights and applying rotary embeddings, the implementation applies a learned *gate* to the output before the final projection back to `d_model` dimensions, allowing the model to modulate attention contributions per layer.

## Advanced Memory and Prediction Features

### Engram KV Cache System

A distinctive feature of this architecture is the **Engram** system—learned hash-based memory modules that augment standard key-value caching. When layers are specified in `config.engram_layers`, the model constructs `Engram` instances at lines 90‑96.

During the forward pass, the `_engram_kv` method (lines 13‑18) builds token-wise indices (`engram_indices`) and masks to query these memory stores, retrieving key-value pairs that capture n-gram statistics. This mechanism allows the model to exploit repetitive patterns across sequences beyond the standard context window.

### Multi-Token Prediction Path

The SAN supports an auxiliary **Multi-Token Prediction (MTP)** branch activated via `return_mtp=True`. Implemented at lines 31‑36, this path processes the next-token embedding through `mtp_emb_norm`, concatenates it with the main stack output, and applies a linear combination via `mtp_combine`. The combined representation passes through a dedicated `mtp_block` and final normalization (`mtp_final_norm`) to produce secondary logits (`mtp_logits`), enabling joint prediction objectives during training.

## Output Generation and Auxiliary Heads

### Causal Masking and Next-Token Projection

The architecture enforces causal constraints through `make_causal_mask` at lines 88‑95, optionally combined with padding masks to prevent future-token leakage. Final vocabulary projection occurs via weight tying: `logits = x @ self.embedding.embedding.T` (lines 27‑28), where the hidden state multiplies the transposed embedding matrix.

### Contrastive and Confidence Heads

For representation learning tasks, the model includes `ContrastiveHead` (lines 43‑60) and `ConfidenceHead` (lines 62‑76). These operate on `hidden_cells`—per-layer hidden states extracted via the method at lines 41‑56. The `hidden_cells` method returns embedding plus all intermediate layer outputs, optionally masked by sliding windows or "sink" patterns, providing rich representations for downstream tasks.

## Implementation Example

The following example demonstrates instantiating the Simple Attention Network and running inference:

```python
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# Minimal configuration (customize all fields as needed)

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=512,
    num_heads=8,
    num_kv_heads=4,
    num_layers=12,
    max_seq_len=1024,
    dtype="bfloat16",
)

model = SimpleAttentionNetwork(config=cfg)

# Dummy input (batch=1, seq_len=10)

import jax.numpy as jnp
tokens = jnp.arange(10)[None, :]          # shape (1, 10)

# Forward pass → logits over the vocab

logits = model(tokens)                    # shape (1, 10, vocab_size)

```

To utilize the Multi-Token Prediction path:

```python
logits, mtp_logits = model(tokens, return_mtp=True)

```

For extracting contrastive embeddings:

```python
query_emb = model.encode_contrastive(tokens)

```

For quantization-aware inference:

```python
logits = model(tokens, quant=True)        # quant-aware inference

```

## Summary

- The **Simple Attention Network** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) extends standard transformers with **Engram memory caching** for n-gram statistics and **Hadamard-MLP** layers for efficient feed-forward processing.
- **Rotary Positional Embeddings** are pre-computed and applied within the `MultiHeadAttention` mechanism to provide relative positional information.
- The architecture supports **Multi-Token Prediction** via an auxiliary branch that processes next-token embeddings in parallel with the main output.
- Optional **ContrastiveHead** and **ConfidenceHead** modules operate on intermediate hidden states extracted via the `hidden_cells` method for downstream representation learning tasks.
- The implementation leverages JAX/Flax primitives including `nn.scan` for memory-efficient layer looping and weight tying for parameter-efficient vocabulary projection.

## Frequently Asked Questions

### What is the Engram KV cache in Needle 2?

The **Engram KV cache** is a learned hash-based memory system that supplements standard key-value caching in specific transformer layers. According to [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), it constructs token-wise indices and masks (lines 13‑18) to store and retrieve n-gram statistics, allowing the model to recognize and exploit repetitive patterns across sequences more effectively than standard attention alone.

### How does Multi-Token Prediction work in the Simple Attention Network?

**Multi-Token Prediction (MTP)** creates an auxiliary prediction branch that forecasts multiple future tokens simultaneously. In the SAN implementation (lines 31‑36), the next-token embedding is normalized, concatenated with the main stack output, and processed through a dedicated `mtp_block` to generate secondary `mtp_logits`, which improves training efficiency and can enhance generation quality through joint loss objectives.

### What is Hadamard-MLP and why is it used?

**Hadamard-MLP** is a feed-forward layer implemented within each `Block` (lines 6‑34) that utilizes Hadamard matrix transformations—orthogonal matrices consisting of ±1 entries. This design reduces computational complexity compared to dense MLP layers while maintaining expressive power, contributing to the "needle-style" efficiency mentioned in the architecture's design philosophy.

### How does the Simple Attention Network handle positional encoding?

The SAN implements **Rotary Positional Embeddings (RoPE)** through the `_rope` helper function at lines 104‑108. Sinusoidal frequencies are pre-computed for each attention head dimension and applied to queries and keys during the forward pass, encoding relative positional information without requiring additional trainable parameters or absolute position embeddings.