# Simple Attention Network (SAN) Architecture in Needle 2: A Complete Technical Breakdown

> Explore the Simple Attention Network (SAN) architecture in Needle 2. Understand its multi-head self-attention, rotary embeddings, and Engram cache for advanced language modeling.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-09

---

**The Simple Attention Network (SAN) is a modular Transformer-style architecture that serves as the core language model in Needle 2, implementing multi-head self-attention with rotary positional embeddings, a learnable Engram key-value cache, and specialized auxiliary heads for contrastive learning and confidence scoring.**

The Simple Attention Network (SAN) architecture powers the Needle 2 framework developed by cactus-compute. Built entirely in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) using Flax `nn.Module` components and JAX primitives, this design prioritizes composability and efficiency through discrete building blocks that support quantization, gradient checkpointing, and long-context modeling.

## Core Components of the SAN Architecture

The SAN implementation follows a layered design pattern where each component handles specific aspects of the transformation pipeline. According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 78-141), the architecture assembles these modules into a cohesive forward pass.

### SimpleAttentionNetwork (Top-Level Module)

The **`SimpleAttentionNetwork`** class acts as the primary entry point for the model. It orchestrates token embedding, positional encoding, and the stacked transformer layers while exposing interfaces for auxiliary tasks. This module manages embedding scaling (multiplying by √`d_model`), pre-computes rotary positional embeddings (ROPE), and initializes the `Stack` of transformer blocks.

### Stack and Block Layers

The **`Stack`** module scans a sequence of **`Block`** layers with optional rematerialization support for memory efficiency. Each `Block` represents a single transformer layer containing:

- **Multi-head self-attention** via the `MultiHeadAttention` module
- **HadamardMLP** for feed-forward processing with learned gating
- **Residual connections** and normalization through utility layers like `ZCRMSNorm`

The `Stack` aggregates outputs from each block and can inject external key-value contexts from the Engram cache.

### MultiHeadAttention with ROPE

The **`MultiHeadAttention`** module implements standard Q-K-V attention with optimizations for modern hardware. It supports optional **flash attention** for memory efficiency and utilizes **rotary position embeddings (ROPE)** to encode positional information directly into the attention mechanism rather than adding separate positional vectors.

### Engram Key-Value Cache

The **`Engram`** module provides a learned key-value cache that stores token n-gram representations. When enabled via `self._engram_kv`, this cache supplies additional context vectors to the attention mechanism, specifically designed to enhance long-range dependency modeling beyond the standard self-attention window.

### Auxiliary Heads: ContrastiveHead and ConfidenceHead

Beyond standard language modeling, the SAN exposes two specialized heads:

- **`ContrastiveHead`**: Generates embeddings for self-supervised contrastive learning tasks
- **`ConfidenceHead`**: Produces scalar confidence scores for each token position, useful for uncertainty quantification during inference

## Data Flow Through the Architecture

The SAN processes input tensors through a six-stage pipeline defined in the `SimpleAttentionNetwork` class definition:

1. **Embedding and scaling** – Input token IDs pass through `nn.Embed` and are scaled by the square root of the model dimension (`self.embed_scale`)
2. **ROPE computation** – Rotary positional encodings are pre-calculated for the current sequence length (`_rope`)
3. **Engram initialization** – If enabled, the Engram builds key/value tables from token n-grams before layer processing begins
4. **Stacked transformation** – The `Stack` module iterates through each `Block`, optionally injecting Engram KV vectors into the attention computation
5. **Logit projection** – Final hidden states project back onto the token embedding matrix to produce vocabulary logits
6. **MTP pathway** – When `return_mtp` is set or during initialization, a multi-token prediction pathway (`mtp_*` modules) executes a second forward pass over the hidden representation for enhanced next-token prediction

## Implementation Details and Quantization

The Needle 2 SAN implementation leverages **Flax** for neural network primitives and **JAX** for automatic differentiation and hardware acceleration. The architecture includes native support for quantization through utilities defined in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py):

- **`_aq`**: Internal quantization helper for activations
- **`maybe_quant_kv`**: Conditional quantization for key-value caches, reducing memory footprint during inference

These utilities integrate directly with the `MultiHeadAttention` and `Engram` modules, allowing the SAN to operate efficiently in low-precision modes without separate conversion steps.

## Working with the Simple Attention Network

The following examples demonstrate practical usage patterns for the SAN architecture using the actual API surface from [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

### Instantiating the Model

Create a SAN instance using the built-in "needle" preset configuration:

```python
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(**SimpleAttentionNetwork.PRESETS["needle"])
model = SimpleAttentionNetwork(cfg)

```

### Running Inference

Execute a forward pass to obtain logits for a batch of token sequences:

```python
import jax.numpy as jnp

tokens = jnp.array([[1, 5, 23, 0, 0]])          # shape: (batch, seq_len)

logits = model(tokens)                          # shape: (batch, seq_len, vocab_size)

```

### Extracting Hidden States

Access intermediate representations for downstream tasks or analysis:

```python
hidden = model.hidden_states(tokens)            # (num_layers, batch, seq_len, d_model)

contrastive_emb = model.encode_contrastive(tokens)

```

### Confidence Scoring

Generate per-token confidence scores using the dedicated head:

```python
confidence = model.forward_confidence(tokens)   # shape: (batch, seq_len)

```

## Summary

- The **Simple Attention Network** in Needle 2 implements a modular Transformer architecture in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), spanning lines 78-141.
- Core components include the **`Stack`** of **`Block`** layers, **`MultiHeadAttention`** with ROPE, and the learnable **`Engram`** key-value cache.
- The architecture supports **quantization-aware** computation through JAX primitives and specialized utilities like `maybe_quant_kv`.
- Auxiliary capabilities include **contrastive learning** and **confidence estimation** via dedicated heads.
- All modules are built on **Flax** and designed for efficient execution with optional gradient rematerialization and flash attention.

## Frequently Asked Questions

### What is the difference between the Stack and Block components in Needle 2's SAN?

The **`Stack`** is a container module that manages the iteration over multiple transformer layers with optional rematerialization for memory efficiency. A **`Block`** represents a single transformer layer containing the multi-head attention mechanism and HadamardMLP sublayers. The Stack handles the sequential application of Blocks across the model depth while managing the Engram cache injection.

### How does the Engram cache improve long-range dependencies?

The **`Engram`** module maintains a learned key-value cache derived from token n-grams that operates independently of the standard self-attention mechanism. By injecting these cached representations into the attention computation within the `Stack`, the model gains access to compressed historical context beyond the immediate sequence window, specifically targeting long-range dependency patterns that standard self-attention might miss.

### What is the purpose of the MTP pathway in the Simple Attention Network?

The **Multi-Token Prediction (MTP)** pathway, activated via `return_mtp` or during model initialization, runs a secondary forward pass over the hidden representations using specialized `mtp_*` modules. This auxiliary pathway enhances the model's ability to predict multiple future tokens simultaneously, providing additional training signal and improved generation quality compared to single-token prediction alone.

### How does Needle 2 handle quantization in the SAN architecture?

Quantization in the SAN is implemented through JAX-compatible utilities in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), specifically the `_aq` helper for activation quantization and `maybe_quant_kv` for conditional key-value cache compression. These functions integrate directly into the `MultiHeadAttention` and `Engram` forward passes, enabling dynamic precision reduction without requiring static model conversion or separate quantization-aware training pipelines.