# Simple Attention Network Architecture: A Technical Deep Dive into Needle's Transformer Implementation

> Explore Needle's Simple Attention Network, a transformer encoder with Engram memory and Multi-Token Prediction. Understand its architecture for contrastive learning and confidence scoring.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-28

---

**The Simple Attention Network is a transformer-style encoder implemented in the cactus-compute/needle repository that combines standard multi-head attention with novel components including Engram memory lookup tables, Multi-Token Prediction pathways, and dual auxiliary heads for contrastive learning and confidence scoring.**

The Simple Attention Network serves as the core model architecture in the cactus-compute/needle repository, extending traditional transformer encoders with specialized memory mechanisms and auxiliary prediction pathways. This architecture integrates efficient attention patterns with learned n-gram statistics through its unique Engram memory system, while supporting advanced training objectives via contrastive and confidence prediction heads.

## Core Components of the Simple Attention Network

The architecture is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and implements a modular design through its `setup` method, configuring distinct subsystems for embedding, transformation, and specialized prediction tasks.

### Token Embeddings and Scaling

The input processing begins with an **embedding layer** that maps token IDs to dense vectors. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines 83-84, the model initializes `self.embedding` and `self.embed_scale`, where the latter implements the standard scaling factor of √d_model to stabilize gradient flow during initial training phases.

### Deep Transformer Stack

At the heart of the network lies a **deep stack of transformer blocks** instantiated as `self.stack = Stack(cfg)` on line 85. This stack processes the embedded sequence through multiple layers of multi-head attention and feed-forward networks, specifically utilizing `HadamardMLP` for efficient transformation.

### Auxiliary Prediction Heads

The architecture features two specialized output heads configured in the `setup` method:

- **Contrastive Head**: Created on lines 86-88 as `self.contrastive_head = ContrastiveHead(cfg.d_model, cfg.contrastive_dim, cfg.jax_dtype)`, this projects hidden states into a lower-dimensional space for contrastive learning objectives.
- **Confidence Head**: Defined on lines 89-90 as `self.confidence_head = ConfidenceHead(cfg.jax_dtype)`, this predicts scalar confidence scores from intermediate hidden states to estimate prediction uncertainty.

### Engram Memory Augmentation

A distinguishing feature of this architecture is the **Engram memory** system—learned lookup tables that provide additional key/value pairs to augment attention mechanisms. Constructed within a list comprehension on lines 92-95, these Engram tables inject learned n-gram statistics into the attention computation through the `_engram_kv` method during forward passes.

### Multi-Token Prediction Pathway

The model supports an auxiliary **Multi-Token Prediction (MTP)** pathway defined on lines 96-102. This subsystem comprises `self.mtp_combine`, `self.mtp_block`, `self.mtp_emb_norm`, and `self.mtp_final_norm`, enabling the model to predict multiple future tokens simultaneously by combining final stack outputs with next-token embeddings.

## Positional Encoding and Masking Strategies

### Rotary Position Embeddings (RoPE)

Rather than traditional absolute positional encodings, the Simple Attention Network implements **RoPE (Rotary Position Embeddings)** through the private `_rope` method on lines 104-108. This method pre-computes cosine and sine frequency matrices that rotate query and key vectors during attention computation, providing better generalization to sequence lengths unseen during training.

### Causal and Padding Mask Utilities

The architecture includes specialized masking utilities to handle variable-length sequences and autoregressive training. Functions such as `make_causal_mask` and `make_padding_mask` are defined within [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) to prevent information leakage from future tokens and to handle padded positions in batched inputs.

## Forward Pass Execution Flow

The `__call__` method orchestrates the following computational graph:

1. **Embedding Phase**: Input tokens are embedded and scaled by the √d_model factor.
2. **Positional Encoding**: The `_rope` method generates rotary frequency matrices.
3. **Memory Augmentation**: Optional Engram key/value pairs are retrieved via `_engram_kv`.
4. **Transformation**: The `Stack` processes the sequence through multiple transformer blocks, returning final hidden states `x`.
5. **Logits Projection**: Language modeling logits are computed by projecting `x` back onto the embedding matrix.
6. **Auxiliary Outputs**: When `return_mtp=True`, the MTP pathway executes by normalizing the next-token embedding, concatenating it with `x`, and processing through the combine layer, block, and final normalization to produce secondary logits.

Additionally, helper methods `encode_contrastive` and `forward_confidence` expose intermediate representations for downstream retrieval and uncertainty quantification tasks.

## Practical Implementation Example

The following example demonstrates instantiation and inference using the Simple Attention Network:

```python
import jax.numpy as jnp
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# 1️⃣ Create a configuration

cfg = TransformerConfig(vocab_size=32000, d_model=768, num_layers=12,
                       num_heads=12, num_kv_heads=6, max_seq_len=2048)

# 2️⃣ Instantiate the model

model = SimpleAttentionNetwork(cfg)

# 3️⃣ Dummy input (batch=2, seq_len=16)

tokens = jnp.ones((2, 16), dtype=jnp.int32)

# 4️⃣ Forward pass – obtain logits

logits = model(tokens)               # shape: (2, 16, vocab_size)

# 5️⃣ Obtain contrastive embeddings

contrastive_emb = model.encode_contrastive(tokens)   # → (2, 16, cfg.contrastive_dim)

# 6️⃣ Get confidence scores

confidence = model.forward_confidence(tokens)       # → (2, 16)

```

## Summary

- **Embedding scaling** utilizes √d_model normalization as implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines 83-84.
- **Transformer stack** processes sequences through `Stack(cfg)` with flash-aware dot-product attention.
- **Engram memory** provides learned n-gram statistics as additional key/value pairs for attention augmentation.
- **Dual auxiliary heads** enable contrastive representation learning and confidence estimation alongside standard language modeling.
- **Multi-Token Prediction** pathway supports auxiliary training objectives by predicting multiple future tokens.
- **RoPE embeddings** replace absolute positional encodings for improved sequence length generalization.

## Frequently Asked Questions

### What makes the Simple Attention Network different from standard transformer architectures?

Unlike conventional transformers that rely solely on self-attention, the Simple Attention Network integrates **Engram memory tables** that supply learned n-gram statistics as supplementary key/value pairs during attention computation. It also incorporates specialized heads for contrastive learning and confidence prediction, alongside a Multi-Token Prediction pathway for auxiliary training signals.

### How does the Engram memory mechanism work in needle?

The Engram memory consists of learned lookup tables constructed in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines 92-95 during the `setup` phase. During the forward pass, the `_engram_kv` method retrieves these learned embeddings to augment the standard key/value cache, effectively allowing the model to attend to statistical n-gram patterns alongside token-level representations.

### What is the purpose of the Multi-Token Prediction pathway?

The **MTP pathway** enables the model to predict multiple future tokens simultaneously by combining final hidden states with next-token embeddings through dedicated projection layers (`self.mtp_combine`, `self.mtp_block`). This auxiliary objective, configured on lines 96-102, provides additional training gradients and can improve sample efficiency during pre-training.

### How are confidence scores computed in this architecture?

Confidence scores are generated by the `ConfidenceHead` instantiated on lines 89-90 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). The `forward_confidence` method processes hidden states through this head to produce scalar values representing the model's uncertainty estimates for each token prediction, enabling applications in selective prediction and uncertainty-aware decoding.