How the Simple Attention Network Architecture Works in Needle 2

The Simple Attention Network (SAN) in Needle 2 replaces traditional transformer feed-forward networks with Hadamard-MLPs, integrates hash-based engram memory, and uses Grouped-Query Attention to deliver high-performance inference with a reduced computational footprint.

The cactus-compute/needle repository implements Needle 2 as a compact yet powerful language model built around the Simple Attention Network architecture. This design eliminates conventional dense layers in favor of orthogonal transforms and adds external n-gram memory through engrams, creating a streamlined alternative to standard transformer stacks.

Model Configuration and Embedding

TransformerConfig Dataclass

All SAN hyperparameters are centralized in the TransformerConfig dataclass defined in needle/model/architecture.py (lines 58-71). This configuration specifies d_model, num_heads, engram_layers, and mhc_lanes, while computing derived values such as attn_dim and JAX dtype settings. The config drives every subsequent layer's dimensions and behavior, ensuring consistency across the embedding table and attention blocks.

Embedding Layer Setup

During SimpleAttentionNetwork.setup(), the model initializes an nn.Embed table of shape vocab_size × d_model and applies the standard "sqrt-dim" scaling trick. Specifically, the embedding weights are multiplied by √d_model to stabilize early training dynamics (lines 83-85). This scaled embedding serves as the initial representation for input tokens before they enter the processing stack.

Core Architectural Components

Engram Memory System

For each layer specified in config.engram_layers, the SAN constructs an Engram instance (lines 90-95). These engrams function as fast key-value tables that store n-gram statistics using hash-based indexing via engram_indices. During the forward pass, engrams provide additional KV pairs to the attention mechanism, effectively augmenting the context window with cheap, external memory that requires minimal compute overhead compared to standard attention caches.

Processing Stack with GQA and Hadamard-MLPs

The heavy lifting occurs within the Stack module (lines 85-86), which scans a _ScanBody containing Block layers across all transformer depths. Each Block contains three critical sub-components:

• Multi-Head GQA Attention (MultiHeadAttention): Implements Grouped-Query Attention with optional Flash-Attention support, reducing KV cache memory bandwidth by sharing key-value heads across query heads.
• Hadamard-MLP: A dense-free feed-forward network that applies an orthogonal Walsh-Hadamard transform (_walsh_matrix) followed by element-wise non-linearities, eliminating learnable weight matrices in the MLP.
• Engram KV Injection: When enabled, engram outputs are concatenated with standard attention KV pairs to enrich the attention context.

Multi-Lane Hyper-Connections (MHC)

After the main stack, the model processes hidden states through a multi-token prediction (MTP) pathway (lines 96-104). This Multi-Lane Hyper-Connections mechanism concatenates the original hidden states with "next-token" embeddings, mixes them via a learnable dense layer (mtp_combine), processes them through an additional Block (mtp_block), and applies a final RMS-norm (mtp_final_norm). This pathway generates richer representations specifically optimized for next-token prediction tasks.

Output Heads and Projections

Logits Projection

The final hidden representation projects back onto the vocabulary through a weight-tied operation using the transpose of the embedding matrix: logits = x @ self.embedding.embedding.T (lines 27-30). The SAN optionally returns both standard logits and MTP logits simultaneously when return_mtp is enabled.

Auxiliary Heads

Two lightweight prediction heads sit atop the hidden cell outputs (lines 44-70):

• ContrastiveHead: Pools cell vectors using learned probes and L2-normalizes them to produce retrieval-style embeddings suitable for contrastive learning tasks.
• ConfidenceHead: Applies similar pooling but outputs a scalar confidence score indicating model certainty for each token position.

Positional Encoding and Optimization

RoPE and Causal Masking

The SAN automatically generates causal attention masks via make_causal_mask (lines 88-90) and supports optional padding masks for batched inference. Rotary Positional Embeddings (RoPE) are pre-computed using precompute_rope_freqs and applied to query and key tensors before attention computation (lines 104-112), allowing the model to generalize to sequence lengths not seen during training.

Quantization Hooks

Throughout the forward pass, the architecture supports optional activation quantization via _aq functions and KV cache quantization through maybe_quant_kv (lines 22-26). These hooks enable 8-bit inference without modifying the model topology or requiring separate quantized checkpoints, reducing memory footprint during deployment.

Running Inference with the Simple Attention Network

The following example demonstrates how to instantiate the SAN, initialize parameters, and utilize the auxiliary heads using the Flax API:

import jax
import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork
from needle.model.tokenizer import SimpleTokenizer

# 1️⃣ Configure and build the model

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    mhc_lanes=4,
    attn_dim=0,          # 0 → use d_model

    flash=True,
    dtype="bfloat16",
)
model = SimpleAttentionNetwork(cfg)

# 2️⃣ Tokenize input

tok = SimpleTokenizer()
tokens = jnp.array([tok.encode("Hello world!")])   # shape (1, N)

# 3️⃣ Initialize and run forward pass

variables = model.init(jax.random.PRNGKey(0), tokens)
logits = model.apply(variables, tokens)            # shape (1, N, vocab_size)

# 4️⃣ Extract hidden states for analysis

hidden = model.apply(
    variables, tokens, 
    method=SimpleAttentionNetwork.hidden_states
)  # shape: (num_layers, batch, seq_len, d_model)

# 5️⃣ Generate contrastive embeddings

query_emb, _, log_temp = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_contrastive,
    query_tokens=tokens,
    tool_tokens=tokens,
)

# 6️⃣ Obtain confidence scores

conf_score = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_confidence,
)
print("Confidence:", conf_score)

Summary

• The Simple Attention Network in Needle 2 replaces standard dense feed-forward layers with Hadamard-MLPs, using orthogonal Walsh-Hadamard transforms to reduce parameters.
• Engram memory provides cheap external KV pairs through hash-based n-gram tables, augmenting attention at specific layers without proportional compute cost.
• Grouped-Query Attention (GQA) and Flash-Attention support minimize memory bandwidth during inference.
• Multi-Lane Hyper-Connections generate enhanced representations for next-token prediction through an auxiliary processing pathway.
• Built-in quantization hooks (_aq, maybe_quant_kv) enable 8-bit inference without architectural changes.

Frequently Asked Questions

What is the difference between Hadamard-MLP and standard transformer MLPs?

Standard transformers use two dense layers with non-linearities between them, requiring substantial parameter counts and memory bandwidth. The Hadamard-MLP in Needle 2 replaces these dense matrices with fixed orthogonal Walsh-Hadamard transforms (_walsh_matrix) followed by element-wise gates, eliminating learnable weights in the feed-forward path while maintaining expressive power through non-linear transformations.

How does engram memory improve model performance?

Engrams store n-gram statistics as additional key-value pairs outside the standard attention cache. According to the source code in needle/model/architecture.py (lines 90-95), these are constructed only for layers specified in engram_layers and use hash-based indexing to retrieve relevant context. This allows the model to access longer-range dependencies and repetitive patterns without increasing the attention window size or computational complexity of the core attention mechanism.

What are the ContrastiveHead and ConfidenceHead used for?

These auxiliary heads operate on the final hidden cell outputs for downstream tasks beyond next-token prediction. The ContrastiveHead (lines 44-55) generates L2-normalized embeddings suitable for retrieval and contrastive learning, while the ConfidenceHead (lines 56-70) outputs scalar values representing model certainty. Both heads use learned probe pools to aggregate information across sequence positions.

Can the Simple Attention Network run with reduced precision?

Yes. The architecture includes explicit quantization support through maybe_quant_kv for the KV cache and _aq hooks for activations (lines 22-26). These functions enable 8-bit quantization during both training and inference without requiring model topology changes or separate quantized model definitions, significantly reducing memory requirements for deployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}