Simple Attention Network (SAN) Architecture in Needle 2: A Complete Technical Breakdown

The Simple Attention Network (SAN) is a modular Transformer-style architecture that serves as the core language model in Needle 2, implementing multi-head self-attention with rotary positional embeddings, a learnable Engram key-value cache, and specialized auxiliary heads for contrastive learning and confidence scoring.

The Simple Attention Network (SAN) architecture powers the Needle 2 framework developed by cactus-compute. Built entirely in needle/model/architecture.py using Flax nn.Module components and JAX primitives, this design prioritizes composability and efficiency through discrete building blocks that support quantization, gradient checkpointing, and long-context modeling.

Core Components of the SAN Architecture

The SAN implementation follows a layered design pattern where each component handles specific aspects of the transformation pipeline. According to the source code in needle/model/architecture.py (lines 78-141), the architecture assembles these modules into a cohesive forward pass.

SimpleAttentionNetwork (Top-Level Module)

The SimpleAttentionNetwork class acts as the primary entry point for the model. It orchestrates token embedding, positional encoding, and the stacked transformer layers while exposing interfaces for auxiliary tasks. This module manages embedding scaling (multiplying by √d_model), pre-computes rotary positional embeddings (ROPE), and initializes the Stack of transformer blocks.

Stack and Block Layers

The Stack module scans a sequence of Block layers with optional rematerialization support for memory efficiency. Each Block represents a single transformer layer containing:

• Multi-head self-attention via the MultiHeadAttention module
• HadamardMLP for feed-forward processing with learned gating
• Residual connections and normalization through utility layers like ZCRMSNorm

The Stack aggregates outputs from each block and can inject external key-value contexts from the Engram cache.

MultiHeadAttention with ROPE

The MultiHeadAttention module implements standard Q-K-V attention with optimizations for modern hardware. It supports optional flash attention for memory efficiency and utilizes rotary position embeddings (ROPE) to encode positional information directly into the attention mechanism rather than adding separate positional vectors.

Engram Key-Value Cache

The Engram module provides a learned key-value cache that stores token n-gram representations. When enabled via self._engram_kv, this cache supplies additional context vectors to the attention mechanism, specifically designed to enhance long-range dependency modeling beyond the standard self-attention window.

Auxiliary Heads: ContrastiveHead and ConfidenceHead

Beyond standard language modeling, the SAN exposes two specialized heads:

• ContrastiveHead: Generates embeddings for self-supervised contrastive learning tasks
• ConfidenceHead: Produces scalar confidence scores for each token position, useful for uncertainty quantification during inference

Data Flow Through the Architecture

The SAN processes input tensors through a six-stage pipeline defined in the SimpleAttentionNetwork class definition:

1. Embedding and scaling – Input token IDs pass through nn.Embed and are scaled by the square root of the model dimension (self.embed_scale)
2. ROPE computation – Rotary positional encodings are pre-calculated for the current sequence length (_rope)
3. Engram initialization – If enabled, the Engram builds key/value tables from token n-grams before layer processing begins
4. Stacked transformation – The Stack module iterates through each Block, optionally injecting Engram KV vectors into the attention computation
5. Logit projection – Final hidden states project back onto the token embedding matrix to produce vocabulary logits
6. MTP pathway – When return_mtp is set or during initialization, a multi-token prediction pathway (mtp_* modules) executes a second forward pass over the hidden representation for enhanced next-token prediction

Implementation Details and Quantization

The Needle 2 SAN implementation leverages Flax for neural network primitives and JAX for automatic differentiation and hardware acceleration. The architecture includes native support for quantization through utilities defined in needle/model/quantize.py:

• _aq: Internal quantization helper for activations
• maybe_quant_kv: Conditional quantization for key-value caches, reducing memory footprint during inference

These utilities integrate directly with the MultiHeadAttention and Engram modules, allowing the SAN to operate efficiently in low-precision modes without separate conversion steps.

Working with the Simple Attention Network

The following examples demonstrate practical usage patterns for the SAN architecture using the actual API surface from needle/model/architecture.py.

Instantiating the Model

Create a SAN instance using the built-in "needle" preset configuration:

from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(**SimpleAttentionNetwork.PRESETS["needle"])
model = SimpleAttentionNetwork(cfg)

Running Inference

Execute a forward pass to obtain logits for a batch of token sequences:

import jax.numpy as jnp

tokens = jnp.array([[1, 5, 23, 0, 0]])          # shape: (batch, seq_len)

logits = model(tokens)                          # shape: (batch, seq_len, vocab_size)

Extracting Hidden States

Access intermediate representations for downstream tasks or analysis:

hidden = model.hidden_states(tokens)            # (num_layers, batch, seq_len, d_model)

contrastive_emb = model.encode_contrastive(tokens)

Confidence Scoring

Generate per-token confidence scores using the dedicated head:

confidence = model.forward_confidence(tokens)   # shape: (batch, seq_len)

Summary

• The Simple Attention Network in Needle 2 implements a modular Transformer architecture in needle/model/architecture.py, spanning lines 78-141.
• Core components include the Stack of Block layers, MultiHeadAttention with ROPE, and the learnable Engram key-value cache.
• The architecture supports quantization-aware computation through JAX primitives and specialized utilities like maybe_quant_kv.
• Auxiliary capabilities include contrastive learning and confidence estimation via dedicated heads.
• All modules are built on Flax and designed for efficient execution with optional gradient rematerialization and flash attention.

Frequently Asked Questions

What is the difference between the Stack and Block components in Needle 2's SAN?

The Stack is a container module that manages the iteration over multiple transformer layers with optional rematerialization for memory efficiency. A Block represents a single transformer layer containing the multi-head attention mechanism and HadamardMLP sublayers. The Stack handles the sequential application of Blocks across the model depth while managing the Engram cache injection.

How does the Engram cache improve long-range dependencies?

The Engram module maintains a learned key-value cache derived from token n-grams that operates independently of the standard self-attention mechanism. By injecting these cached representations into the attention computation within the Stack, the model gains access to compressed historical context beyond the immediate sequence window, specifically targeting long-range dependency patterns that standard self-attention might miss.

What is the purpose of the MTP pathway in the Simple Attention Network?

The Multi-Token Prediction (MTP) pathway, activated via return_mtp or during model initialization, runs a secondary forward pass over the hidden representations using specialized mtp_* modules. This auxiliary pathway enhances the model's ability to predict multiple future tokens simultaneously, providing additional training signal and improved generation quality compared to single-token prediction alone.

How does Needle 2 handle quantization in the SAN architecture?

Quantization in the SAN is implemented through JAX-compatible utilities in needle/model/quantize.py, specifically the _aq helper for activation quantization and maybe_quant_kv for conditional key-value cache compression. These functions integrate directly into the MultiHeadAttention and Engram forward passes, enabling dynamic precision reduction without requiring static model conversion or separate quantization-aware training pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}