How the Simple Attention Network Architecture Works in Needle 2

The Simple Attention Network (SAN) in Needle 2 replaces conventional transformer feed-forward layers with Hadamard-based MLPs and augments standard attention with engram memory buffers and multi-lane hyper-connections to deliver high performance with reduced parameter counts.

The Simple Attention Network architecture lives in the cactus-compute/needle repository as a JAX/Flax implementation designed for efficient training and inference. Unlike standard dense transformers, the SAN eliminates traditional feed-forward weight matrices in favor of orthogonal transforms, integrates external key-value caches through engram layers, and employs grouped-query attention to minimize memory bandwidth. Examining the implementation in needle/model/architecture.py reveals how these components orchestrate the flow from input tokens to final logits.

Core Architecture Components

The SAN is defined by the SimpleAttentionNetwork class, which orchestrates eight distinct processing stages. Each stage is deliberately optimized: the model avoids dense parameters where possible, uses hash-based lookups for n-gram statistics, and provides auxiliary heads for contrastive learning and confidence scoring.

Configuration and Embedding Initialization

Model behavior is governed by the TransformerConfig dataclass (lines 58-71), which specifies hyperparameters including d_model, num_heads, engram_layers, and mhc_lanes. During initialization in SimpleAttentionNetwork.setup(), the architecture creates an nn.Embed table of shape vocab_size × d_model and applies the "sqrt-dim" scaling trick by multiplying embeddings by √d_model (lines 83-85). This scaling stabilizes early training by preventing dot-product attention scores from growing too large.

Engram Memory System

For each layer index listed in config.engram_layers, the model constructs an Engram instance (lines 90-95). These engrams function as fast key-value tables that store n-gram statistics using hash-based indexing (engram_indices). During the forward pass, engrams inject additional KV pairs into the attention computation, effectively giving the model access to external memory that captures multi-token patterns without increasing the parameter count of the main attention layers.

The Processing Stack: Attention and Hadamard-MLP

The heavy computation occurs within the Stack module (line 85-86), which executes a scanned Block (_ScanBody) across all transformer layers. Each Block contains three critical sub-components:

• Multi-Head GQA Attention: Implemented via MultiHeadAttention, this uses Grouped-Query Attention (GQA) to share key/value heads across multiple query heads, reducing KV cache memory requirements. The implementation optionally supports FlashAttention for accelerated computation.
• Hadamard-MLP: Instead of dense linear layers, the feed-forward network applies an orthogonal Walsh-Hadamard transform (_walsh_matrix) followed by element-wise gating non-linearities. This approach eliminates learnable weight matrices in the MLP while maintaining expressiveness through structured transforms.
• Engram KV Injection: When engram layers are active, the block integrates external KV tensors from the engram tables into the attention computation.

Multi-Lane Hyper-Connections (MHC)

Following the main stack, the model processes outputs through a multi-lane pathway defined by mtp_* members (lines 96-104). This pathway:

1. Concatenates original hidden states with "next-token" embeddings
2. Mixes representations via a learnable dense layer (mtp_combine)
3. Processes the mixture through an additional Block (mtp_block)
4. Applies a final RMS normalization (mtp_final_norm)

This architecture provides richer representations for next-token prediction by explicitly modeling future token contexts in a separate computational lane.

Output Projection and Auxiliary Heads

The final hidden representation projects back onto the vocabulary via the transposed embedding matrix: logits = x @ self.embedding.embedding.T (lines 27-30). The model optionally returns both standard and MTP (multi-token prediction) logits.

Two lightweight auxiliary heads operate on hidden cell outputs:

• ContrastiveHead: Pools cell vectors using learned probes and L2-normalizes them to produce retrieval-ready embeddings (lines 44-55)
• ConfidenceHead: Pools hidden cells via learned parameters to output a scalar confidence score for each token prediction (lines 56-70)

Positional Encodings and Causal Masking

The SAN implements Rotary Positional Embeddings (RoPE) through precompute_rope_freqs and applies them to query/key tensors via apply_rope (lines 104-112). Causal masking is generated automatically via make_causal_mask (lines 88-90) to prevent attention to future tokens, with optional support for padding masks and sliding-window attention constraints.

Quantization Support

Throughout the forward pass, the architecture supports optional activation quantization via _aq hooks and KV tensor quantization via maybe_quant_kv (lines 22-26). These mechanisms enable 8-bit inference without modifying the model topology, reducing memory footprint for deployment.

Running the Simple Attention Network

The following example demonstrates instantiating the SAN, processing tokens, and utilizing auxiliary heads:

# 1️⃣ Build the model -------------------------------------------------

import jax
import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    mhc_lanes=4,
    attn_dim=0,          # 0 → use d_model

    flash=True,
    dtype="bfloat16",
)

model = SimpleAttentionNetwork(cfg)

# 2️⃣ Tokenise a dummy input -----------------------------------------

from needle.model.tokenizer import SimpleTokenizer
tok = SimpleTokenizer()
tokens = jnp.array([tok.encode("Hello world!")])   # shape (1, N)

# 3️⃣ Forward pass (logits only) --------------------------------------

variables = model.init(jax.random.PRNGKey(0), tokens)
logits = model.apply(variables, tokens)                # shape (1, N, vocab)

# 4️⃣ Retrieve hidden states (for probing) ---------------------------

hidden = model.apply(
    variables, 
    tokens, 
    method=SimpleAttentionNetwork.hidden_states
)

# hidden shape: (num_layers, batch, seq_len, d_model)

# 5️⃣ Contrastive embedding (useful for retrieval) -------------------

query_emb, _, log_temp = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_contrastive,
    query_tokens=tokens,
    tool_tokens=tokens,
)

# 6️⃣ Confidence score ------------------------------------------------

conf_score = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_confidence,
)
print("Confidence:", conf_score)

The model.init and model.apply methods follow standard Flax patterns. Optional arguments like quant, return_mtp, and window enable 8-bit inference, multi-token prediction outputs, and sliding-window attention respectively.

Summary

• The Simple Attention Network eliminates dense feed-forward parameters by using Hadamard-MLP layers with orthogonal Walsh-Hadamard transforms and element-wise gates.
• Engram memory layers provide external KV caches that inject n-gram statistics into specific attention layers via hash-based lookups.
• Grouped-Query Attention (GQA) reduces KV cache memory bandwidth by sharing key/value heads across query heads.
• Multi-Lane Hyper-Connections process an auxiliary pathway with next-token embeddings to enhance prediction quality.
• Auxiliary heads (Contrastive and Confidence) enable retrieval-based training and uncertainty quantification without external models.
• Full implementation resides in needle/model/architecture.py, with tokenization handled by needle/model/tokenizer.py.

Frequently Asked Questions

What distinguishes the Simple Attention Network from standard transformer architectures?

The SAN replaces the conventional dense feed-forward network with a Hadamard-MLP that uses orthogonal transforms rather than learnable weight matrices, and it augments attention with engram memory layers that provide external KV caches populated by n-gram statistics. According to the cactus-compute/needle source code, these modifications reduce parameter counts while maintaining representational capacity through structured transforms and external memory lookups.

How does engram memory work in Needle 2?

Engrams are implemented as fast key-value tables instantiated for layers specified in config.engram_layers (lines 90-95 of needle/model/architecture.py). They use hash-based indexing (engram_indices) to store and retrieve n-gram statistics, effectively giving the model access to multi-token pattern memory without increasing the trainable parameters of the attention mechanism itself.

What are the Multi-Lane Hyper-Connections (MHC) used for?

The MHC pathway (defined by mtp_* members in lines 96-104) creates a secondary processing lane that concatenates hidden states with next-token embeddings, mixes them through a learnable dense layer (mtp_combine), and processes them through an additional transformer block. This architecture enriches the representation for next-token prediction by explicitly modeling future context in a separate computational stream before final normalization.

How can I extract confidence scores or embeddings from the model?

The SAN provides two dedicated methods: SimpleAttentionNetwork.forward_confidence returns a scalar confidence score via the ConfidenceHead, while SimpleAttentionNetwork.forward_contrastive produces L2-normalized embeddings suitable for retrieval tasks via the ContrastiveHead. Both methods can be invoked through Flax's model.apply by specifying the method parameter, as shown in the implementation example above.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}