Simple Attention Network Architecture in Needle 2: A Complete Technical Guide

The Simple Attention Network (SAN) is the core transformer-style encoder in Needle 2, implemented in needle/model/architecture.py, featuring learned engram memory, multi-token prediction (MTP), and auxiliary contrastive/confidence heads designed for high-performance, on-device inference.

The Simple Attention Network serves as the primary neural architecture powering Needle 2's language modeling capabilities. This lightweight yet expressive model extends standard transformer encoders with specialized components for long-range retrieval and self-supervised training. According to the cactus-compute/needle source code, the SAN implementation emphasizes efficiency through careful attention mechanisms and optional quantization support.

Core Architecture Components

The SAN implementation in needle/model/architecture.py instantiates seven key components during initialization in the setup() method.

Embedding Layer with RMS Scaling

The model begins with a standard token embedding that scales outputs by √d_model. As implemented at line 82, self.embedding maps input tokens to dense vectors before the scaling operation prepares them for the attention mechanism.

Multi-Head Attention Stack

The central processing unit is the Stack class (line 84), which contains multiple attention blocks. This component handles standard multi-head attention with optional flash-attention optimization and KV-budget management for memory-constrained environments.

Engram Memory System

A distinctive feature of the SAN architecture is the engram memory system (line 90). These learned tables store compressed n-gram statistics and supply additional key/value pairs to the attention mechanism. This design enables long-range context retrieval without exhausting the KV-cache budget, effectively extending the model's receptive field beyond standard attention windows.

Multi-Token Prediction Block

The MTP (Multi-Token-Predict) system consists of self.mtp_combine (a dense projection layer) and self.mtp_block (an additional attention block) initialized at line 96. When activated via return_mtp=True, this mechanism concatenates the main hidden state with shifted embeddings, projects them through the dense layer, and processes them through the attention block to improve next-token prediction accuracy.

Contrastive and Confidence Heads

The architecture includes two specialized heads initialized in setup():

• Contrastive Head (line 86): Projects pooled representations to a low-dimensional space for contrastive learning of token-level embeddings
• Confidence Head (line 88): Predicts scalar confidence scores for uncertainty estimation and tool-selection logic

Rotary Positional Embeddings

Rather than learned positional encodings, the SAN uses pre-computed sinusoidal frequencies applied via the _rope helper (line 104) to queries and keys, providing positional information without additional parameters.

Causal and Padding Masks

The forward pass enforces autoregressive behavior through causal masks created by make_causal_mask (line 88) and optional padding masks via make_padding_mask (line 93), ensuring the model only attends to previous positions and properly handles variable-length sequences.

Forward Pass Execution Flow

The __call__ method in SimpleAttentionNetwork orchestrates the following computation graph:

1. Embedding and Scaling: Input tokens pass through self.embedding and multiply by √d_model
2. RoPE Generation: Cosine/sine tables compute for the current sequence length
3. Engram KV Retrieval: Optional extra key/value pairs generated from engram tables at specified layers (2 and 15 by default)
4. Stack Processing: The main self.stack processes the sequence through attention blocks, returning hidden states
5. Logits Projection: Final logits project back onto the token-embedding matrix via weight tying
6. Optional MTP: When return_mtp=True, the mtp_block produces additional prediction logits through the second-stage mechanism

Source Code Structure

The SAN resides in the SimpleAttentionNetwork class defined at line 78 of needle/model/architecture.py. Key implementation files include:

• needle/model/architecture.py: Contains TransformerConfig, SimpleAttentionNetwork, attention blocks, engram memory, and prediction heads
• needle/model/run.py: High-level run helper for model instantiation and inference execution
• needle/model/quantize.py: Fake-quantization implementation for quant=True mode
• tests/test_inference.py: Unit tests validating SAN forward passes and MTP outputs

Implementation Example

The following examples demonstrate typical usage patterns:

# Model instantiation

from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    contrastive_dim=128,
)

model = SimpleAttentionNetwork(config=cfg)

# Standard inference

import jax.numpy as jnp

tokens = jnp.array([[1, 5, 23, 0, 0]])  # batch_size=1, seq_len=5

logits = model(tokens)  # Shape: (1, 5, 8192)

# Multi-token prediction mode

logits, mtp_logits = model(tokens, return_mtp=True)

# mtp_logits shape: (1, 5, 8192)

# Contrastive encoding

embeddings, log_temp = model.encode_contrastive(tokens)

# embeddings shape: (batch, contrastive_dim)

Summary

• The Simple Attention Network in Needle 2 implements a customized transformer encoder in needle/model/architecture.py with specialized components for efficient inference
• Engram memory provides compressed n-gram statistics as additional attention keys/values, extending context windows without KV-cache exhaustion
• The MTP block enables two-stage prediction when return_mtp=True, improving next-token accuracy through additional processing
• Contrastive and confidence heads support self-supervised training and uncertainty quantification respectively
• RoPE embeddings supply positional information without parameter overhead
• The architecture supports optional quantization through needle/model/quantize.py for deployment on resource-constrained devices

Frequently Asked Questions

What makes the Simple Attention Network different from standard transformer architectures?

The SAN extends conventional transformers with learned engram memory tables that compress n-gram statistics and feed them as additional key/value pairs to the attention mechanism. Unlike standard transformers that rely solely on the KV cache, this approach enables longer effective context windows without proportional memory increases. The architecture also includes a Multi-Token Prediction block for improved next-token accuracy and auxiliary heads for contrastive learning and confidence estimation.

How does the engram memory system work in practice?

During the forward pass, specifically at layers 2 and 15 by default, the model retrieves compressed representations from learned engram tables. These representations serve as additional keys and values concatenated with the standard attention inputs. Implemented in self.engrams at line 90 of needle/model/architecture.py, this mechanism allows the model to access long-range statistical patterns without storing full history in the KV cache.

When should I use the Multi-Token Prediction (MTP) feature?

Activate MTP by passing return_mtp=True to the model's __call__ method when you need improved next-token prediction accuracy and can afford the computational cost of the second-stage processing. The MTP block concatenates hidden states with shifted embeddings and runs them through an additional attention block, producing mtp_logits alongside standard outputs. This feature is particularly useful during model initialization or when maximum prediction quality is required.

Where is the Simple Attention Network implemented in the Needle 2 codebase?

The core implementation resides in the SimpleAttentionNetwork class at line 78 of needle/model/architecture.py. This file defines the complete architecture including the Stack class for attention blocks, engram memory systems, and auxiliary heads. Supporting functionality appears in needle/model/run.py for high-level inference APIs and needle/model/quantize.py for optional quantization support.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}