How Needle 2's Simple Attention Network Architecture Works: A Complete Technical Guide

Needle 2's Simple Attention Network (SAN) is a JAX/Flax-based transformer that combines standard multi-head self-attention with novel Engram memory caching, Hadamard-MLP layers, and auxiliary Multi-Token Prediction heads to achieve efficient, scalable language modeling.

The Needle 2 repository implements the Simple Attention Network as its core language model in needle/model/architecture.py. This Needle 2 Simple Attention Network architecture extends the classic transformer stack with specialized optimizations for memory efficiency and multi-task learning while maintaining compatibility with standard attention mechanisms. Below is a comprehensive walkthrough of the SAN pipeline, from token embedding to final logits.

Configuration and Initialization

All hyperparameters governing the Simple Attention Network are centralized in the TransformerConfig dataclass defined at lines 58‑83 of needle/model/architecture.py. This configuration object specifies vocab_size, d_model, num_heads, num_kv_heads, num_layers, and specialized fields like rope_theta and engram_* parameters that control optional memory augmentation.

During initialization, SimpleAttentionNetwork uses this config to instantiate the embedding layer, pre-compute rotary positional encodings, and construct the transformer stack with optional Engram modules. The embedding layer appears in the setup method at lines 78‑85, utilizing nn.Embed with vectors scaled by √d_model for numerical stability.

Core Architecture Components

Rotary Positional Embeddings (RoPE)

The SAN implements sinusoidal rotary positional encoding via the _rope helper function (precompute_rope_freqs) at lines 104‑108. These frequencies are pre-computed for each head dimension and applied to queries and keys within the attention blocks, providing relative positional information without additional learned parameters.

The Stack and Block Structure

The transformer core utilizes a Stack module defined at lines 76‑86 that repeatedly applies Block layers using nn.scan for memory-efficient iteration across num_layers. Each Block (lines 6‑34) executes the following operations:

• Optional Engram injection that augments hidden states with retrieved memory values
• ZCRMSNorm (a specialized LayerNorm variant) followed by MultiHeadAttention
• Residual connection with a learned gating scalar
• Hadamard-MLP, an efficient feed-forward layer utilizing orthogonal matrix transformations

Multi-Head Attention Mechanism

The MultiHeadAttention class at lines 9‑78 handles query, key, and value projections with support for optional Flash Attention. After computing attention weights and applying rotary embeddings, the implementation applies a learned gate to the output before the final projection back to d_model dimensions, allowing the model to modulate attention contributions per layer.

Advanced Memory and Prediction Features

Engram KV Cache System

A distinctive feature of this architecture is the Engram system—learned hash-based memory modules that augment standard key-value caching. When layers are specified in config.engram_layers, the model constructs Engram instances at lines 90‑96.

During the forward pass, the _engram_kv method (lines 13‑18) builds token-wise indices (engram_indices) and masks to query these memory stores, retrieving key-value pairs that capture n-gram statistics. This mechanism allows the model to exploit repetitive patterns across sequences beyond the standard context window.

Multi-Token Prediction Path

The SAN supports an auxiliary Multi-Token Prediction (MTP) branch activated via return_mtp=True. Implemented at lines 31‑36, this path processes the next-token embedding through mtp_emb_norm, concatenates it with the main stack output, and applies a linear combination via mtp_combine. The combined representation passes through a dedicated mtp_block and final normalization (mtp_final_norm) to produce secondary logits (mtp_logits), enabling joint prediction objectives during training.

Output Generation and Auxiliary Heads

Causal Masking and Next-Token Projection

The architecture enforces causal constraints through make_causal_mask at lines 88‑95, optionally combined with padding masks to prevent future-token leakage. Final vocabulary projection occurs via weight tying: logits = x @ self.embedding.embedding.T (lines 27‑28), where the hidden state multiplies the transposed embedding matrix.

Contrastive and Confidence Heads

For representation learning tasks, the model includes ContrastiveHead (lines 43‑60) and ConfidenceHead (lines 62‑76). These operate on hidden_cells—per-layer hidden states extracted via the method at lines 41‑56. The hidden_cells method returns embedding plus all intermediate layer outputs, optionally masked by sliding windows or "sink" patterns, providing rich representations for downstream tasks.

Implementation Example

The following example demonstrates instantiating the Simple Attention Network and running inference:

from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# Minimal configuration (customize all fields as needed)

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=512,
    num_heads=8,
    num_kv_heads=4,
    num_layers=12,
    max_seq_len=1024,
    dtype="bfloat16",
)

model = SimpleAttentionNetwork(config=cfg)

# Dummy input (batch=1, seq_len=10)

import jax.numpy as jnp
tokens = jnp.arange(10)[None, :]          # shape (1, 10)

# Forward pass → logits over the vocab

logits = model(tokens)                    # shape (1, 10, vocab_size)

To utilize the Multi-Token Prediction path:

logits, mtp_logits = model(tokens, return_mtp=True)

For extracting contrastive embeddings:

query_emb = model.encode_contrastive(tokens)

For quantization-aware inference:

logits = model(tokens, quant=True)        # quant-aware inference

Summary

• The Simple Attention Network in needle/model/architecture.py extends standard transformers with Engram memory caching for n-gram statistics and Hadamard-MLP layers for efficient feed-forward processing.
• Rotary Positional Embeddings are pre-computed and applied within the MultiHeadAttention mechanism to provide relative positional information.
• The architecture supports Multi-Token Prediction via an auxiliary branch that processes next-token embeddings in parallel with the main output.
• Optional ContrastiveHead and ConfidenceHead modules operate on intermediate hidden states extracted via the hidden_cells method for downstream representation learning tasks.
• The implementation leverages JAX/Flax primitives including nn.scan for memory-efficient layer looping and weight tying for parameter-efficient vocabulary projection.

Frequently Asked Questions

What is the Engram KV cache in Needle 2?

The Engram KV cache is a learned hash-based memory system that supplements standard key-value caching in specific transformer layers. According to needle/model/architecture.py, it constructs token-wise indices and masks (lines 13‑18) to store and retrieve n-gram statistics, allowing the model to recognize and exploit repetitive patterns across sequences more effectively than standard attention alone.

How does Multi-Token Prediction work in the Simple Attention Network?

Multi-Token Prediction (MTP) creates an auxiliary prediction branch that forecasts multiple future tokens simultaneously. In the SAN implementation (lines 31‑36), the next-token embedding is normalized, concatenated with the main stack output, and processed through a dedicated mtp_block to generate secondary mtp_logits, which improves training efficiency and can enhance generation quality through joint loss objectives.

What is Hadamard-MLP and why is it used?

Hadamard-MLP is a feed-forward layer implemented within each Block (lines 6‑34) that utilizes Hadamard matrix transformations—orthogonal matrices consisting of ±1 entries. This design reduces computational complexity compared to dense MLP layers while maintaining expressive power, contributing to the "needle-style" efficiency mentioned in the architecture's design philosophy.

How does the Simple Attention Network handle positional encoding?

The SAN implements Rotary Positional Embeddings (RoPE) through the _rope helper function at lines 104‑108. Sinusoidal frequencies are pre-computed for each attention head dimension and applied to queries and keys during the forward pass, encoding relative positional information without requiring additional trainable parameters or absolute position embeddings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}