Simple Attention Network (SAN) Architecture in Needle: A Technical Deep Dive

The Simple Attention Network (SAN) is a Transformer-style encoder-decoder that augments standard multi-head attention with a learned Engram memory, Hadamard MLPs, and auxiliary heads for contrastive learning and confidence estimation, implemented in the SimpleAttentionNetwork class in needle/model/architecture.py.

The Simple Attention Network (SAN) serves as the core neural architecture powering Needle, an open-source inference framework maintained by cactus-compute. Unlike standard Transformer implementations, SAN integrates associative memory mechanisms and task-specific prediction heads directly into its forward pass. This guide examines the complete SAN implementation by tracing its components from embedding initialization to autoregressive generation.

Core Architecture Components

Token Embedding and Scaling

During initialization in setup(), SAN creates an embedding matrix using Flax's nn.Embed configured with a standard deviation of 0.02. The implementation in needle/model/architecture.py (around line 83) scales token embeddings by the square root of the model dimension to preserve variance across depth:

self.embedding = nn.Embed(cfg.vocab_size, cfg.d_model, embedding_init=jinit.normal(stddev=0.02))
self.embed_scale = math.sqrt(cfg.d_model)

Input tokens are embedded and multiplied by embed_scale before entering the main transformer stack.

The Stack and Transformer Blocks

The Stack module, defined at line 76 of architecture.py, encapsulates the full sequence of transformer layers. Each layer is a Block containing:

• Multi-Head Attention (MultiHeadAttention) with optional flash-attention support for GPU acceleration
• Hadamard MLP (HadamardMLP) replacing standard feed-forward networks with Walsh-Hadamard transforms for computational efficiency
• Layer-wise gating to regulate residual contributions

The stack leverages nn.scan or nn.remat (when remat=True) to enable efficient JIT compilation and reversible memory checkpointing.

Engram Memory System

SAN optionally incorporates Engram memory, a learned associative memory mechanism defined at line 81 of architecture.py. The _engram_kv method computes hashed n-gram indices and retrieves key/value tables that fuse with standard attention KV caches. This design provides sub-linear scaling with sequence length by augmenting the context with hashed lookups rather than full attention over all prior tokens.

Auxiliary Prediction Heads

Beyond the main language modeling head, SAN computes three auxiliary outputs:

• ContrastiveHead (line 44): Projects hidden states to a low-dimensional space (cfg.contrastive_dim) with learned temperature parameters for retrieval and ranking tasks
• ConfidenceHead (line 64): Predicts scalar confidence scores for generated sequences using probe pooling mechanisms
• MTP (Multi-Task Prediction) block (lines 99-101): An auxiliary pathway that fuses final hidden states with next-step token embeddings to produce secondary logits for tool-calling or structured output generation

Configuration and Initialization

SAN receives hyperparameters through the TransformerConfig dataclass defined at line 58 of architecture.py. Key configuration fields include:

• d_model: Hidden dimension size
• num_heads / num_kv_heads: Attention head configuration (supports GQA)
• num_layers: Depth of the transformer stack
• vocab_size: Token vocabulary dimensions
• max_seq_len: Maximum context window
• engram_layers, engram_orders, engram_heads, engram_slots: Controls for the Engram memory system

The quant flag enables fake quantization throughout the model for training-time approximation of low-bit inference, utilizing helpers from needle/model/quantize.py such as fake_quant_act and maybe_quant_kv.

The Forward Pass Execution Flow

The __call__ method of SimpleAttentionNetwork executes the following sequence:

1. Mask Creation: Generates a causal attention mask if none is provided
2. Embedding: Scales input token embeddings by sqrt(d_model)
3. Positional Encoding: Pre-computes Rotary Positional Embeddings (RoPE) via precompute_rope_freqs (line 97)
4. Engram Retrieval: Optionally computes Engram KV pairs through _engram_kv
5. Stack Processing: Passes embeddings through the transformer stack with attention and MLP blocks
6. Logit Projection: Projects final hidden states back to vocabulary space using the transposed embedding matrix
7. MTP Pathway: If return_mtp=True, computes secondary logits through the MTP block

All operations respect the quantization configuration when quant is enabled.

Generating Text with SAN: Practical Example

The following example demonstrates loading a checkpoint, instantiating SAN, and generating text. This mirrors the CLI implementation in needle/model/run.py:

import pathlib
from needle.model.run import load_checkpoint, generate
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig
from needle.model.tokenizer import get_tokenizer

# 1️⃣ Load a checkpoint (downloaded automatically if missing)

ckpt_path = pathlib.Path("checkpoints/needle_v1.ckpt")
params, cfg = load_checkpoint(ckpt_path)

# 2️⃣ Instantiate the model

model = SimpleAttentionNetwork(cfg)

# 3️⃣ Retrieve the tokenizer (downloads from HF if needed)

tokenizer = get_tokenizer(cfg.vocab_size)

# 4️⃣ Generate a continuation

prompt = "The most surprising thing about"
output = generate(
    model=model,
    params=params,
    tokenizer=tokenizer,
    prompt=prompt,
    max_new_tokens=64,
    temperature=0.0,      # deterministic sampling

    seed=42,
    stream=False,        # returns the full text at once

)

print("🖋️", output)

Key functions referenced:

• load_checkpoint: Loads serialized parameters from line 62 of run.py
• get_tokenizer: Returns a SentencePiece tokenizer from line 35 of tokenizer.py
• generate: Autoregressive sampling loop defined at line 77 of run.py

Advanced Capabilities

SAN exposes several specialized methods for downstream tasks:

• Contrastive encoding: model.encode_contrastive(tokens) returns normalized representations for semantic search
• Confidence scoring: model.forward_confidence(tokens) yields per-sequence uncertainty estimates
• Hidden state extraction: model.hidden_cells(tokens, window=128) returns per-layer activations for analysis or fine-tuning workflows

Summary

• Simple Attention Network (SAN) combines standard Transformer blocks with novel components including Engram memory and Hadamard MLPs in needle/model/architecture.py
• Configuration is managed through the TransformerConfig dataclass (line 58), controlling model dimensions, attention heads, and memory parameters
• Engram memory provides hashed associative lookup capabilities that augment standard KV caches, scaling efficiently with sequence length
• Auxiliary heads enable contrastive learning, confidence calibration, and multi-task prediction without requiring separate model instances
• Quantization support is built into the forward pass, allowing training-time simulation of low-bit inference using utilities from quantize.py

Frequently Asked Questions

What distinguishes SAN from standard Transformer architectures?

SAN extends the traditional Transformer encoder-decoder by integrating an Engram memory system for hashed n-gram lookups and replacing standard feed-forward layers with Hadamard MLPs for computational efficiency. These modifications, located in needle/model/architecture.py, enable better scaling properties and auxiliary task capabilities while maintaining compatibility with standard attention mechanisms.

How does the Engram memory system work?

The Engram system computes hashed indices for input sequences via the _engram_kv method and retrieves learned key/value pairs that fuse with standard attention KV caches. This associative memory mechanism allows the model to access relevant context through hash-based lookups rather than full attention computation, reducing memory complexity as sequence length increases.

What are the auxiliary heads used for in SAN?

SAN includes three specialized heads: the ContrastiveHead projects representations for retrieval and ranking tasks, the ConfidenceHead estimates prediction uncertainty, and the MTP block generates secondary logits for tool-calling or structured outputs. These heads share probe pooling mechanisms and operate on the final hidden states from the main transformer stack.

How do I configure and instantiate a SAN model?

Create a TransformerConfig dataclass instance specifying d_model, num_heads, num_layers, and Engram parameters, then pass it to SimpleAttentionNetwork(cfg) as shown in needle/model/architecture.py line 78. Load parameters using load_checkpoint() from run.py (line 62) and generate text via the generate() function with appropriate tokenizer configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}