# How the Simple Attention Network Architecture Works in Needle 2

> Discover how Needle 2's Simple Attention Network architecture boosts performance with Hadamard MLPs, engram memory buffers, and multi-lane hyper-connections, all while reducing parameter counts.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-10

---

**The Simple Attention Network (SAN) in Needle 2 replaces conventional transformer feed-forward layers with Hadamard-based MLPs and augments standard attention with engram memory buffers and multi-lane hyper-connections to deliver high performance with reduced parameter counts.**

The Simple Attention Network architecture lives in the `cactus-compute/needle` repository as a JAX/Flax implementation designed for efficient training and inference. Unlike standard dense transformers, the SAN eliminates traditional feed-forward weight matrices in favor of orthogonal transforms, integrates external key-value caches through engram layers, and employs grouped-query attention to minimize memory bandwidth. Examining the implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) reveals how these components orchestrate the flow from input tokens to final logits.

## Core Architecture Components

The SAN is defined by the `SimpleAttentionNetwork` class, which orchestrates eight distinct processing stages. Each stage is deliberately optimized: the model avoids dense parameters where possible, uses hash-based lookups for n-gram statistics, and provides auxiliary heads for contrastive learning and confidence scoring.

## Configuration and Embedding Initialization

Model behavior is governed by the `TransformerConfig` dataclass (lines 58-71), which specifies hyperparameters including `d_model`, `num_heads`, `engram_layers`, and `mhc_lanes`. During initialization in `SimpleAttentionNetwork.setup()`, the architecture creates an `nn.Embed` table of shape `vocab_size × d_model` and applies the "sqrt-dim" scaling trick by multiplying embeddings by √`d_model` (lines 83-85). This scaling stabilizes early training by preventing dot-product attention scores from growing too large.

## Engram Memory System

For each layer index listed in `config.engram_layers`, the model constructs an `Engram` instance (lines 90-95). These engrams function as fast key-value tables that store n-gram statistics using hash-based indexing (`engram_indices`). During the forward pass, engrams inject additional KV pairs into the attention computation, effectively giving the model access to external memory that captures multi-token patterns without increasing the parameter count of the main attention layers.

## The Processing Stack: Attention and Hadamard-MLP

The heavy computation occurs within the `Stack` module (line 85-86), which executes a scanned `Block` (`_ScanBody`) across all transformer layers. Each `Block` contains three critical sub-components:

- **Multi-Head GQA Attention**: Implemented via `MultiHeadAttention`, this uses **Grouped-Query Attention (GQA)** to share key/value heads across multiple query heads, reducing KV cache memory requirements. The implementation optionally supports FlashAttention for accelerated computation.
- **Hadamard-MLP**: Instead of dense linear layers, the feed-forward network applies an orthogonal Walsh-Hadamard transform (`_walsh_matrix`) followed by element-wise gating non-linearities. This approach eliminates learnable weight matrices in the MLP while maintaining expressiveness through structured transforms.
- **Engram KV Injection**: When engram layers are active, the block integrates external KV tensors from the engram tables into the attention computation.

## Multi-Lane Hyper-Connections (MHC)

Following the main stack, the model processes outputs through a multi-lane pathway defined by `mtp_*` members (lines 96-104). This pathway:
1. Concatenates original hidden states with "next-token" embeddings
2. Mixes representations via a learnable dense layer (`mtp_combine`)
3. Processes the mixture through an additional `Block` (`mtp_block`)
4. Applies a final RMS normalization (`mtp_final_norm`)

This architecture provides richer representations for next-token prediction by explicitly modeling future token contexts in a separate computational lane.

## Output Projection and Auxiliary Heads

The final hidden representation projects back onto the vocabulary via the transposed embedding matrix: `logits = x @ self.embedding.embedding.T` (lines 27-30). The model optionally returns both standard and MTP (multi-token prediction) logits.

Two lightweight auxiliary heads operate on hidden cell outputs:

- **ContrastiveHead**: Pools cell vectors using learned probes and L2-normalizes them to produce retrieval-ready embeddings (lines 44-55)
- **ConfidenceHead**: Pools hidden cells via learned parameters to output a scalar confidence score for each token prediction (lines 56-70)

## Positional Encodings and Causal Masking

The SAN implements Rotary Positional Embeddings (RoPE) through `precompute_rope_freqs` and applies them to query/key tensors via `apply_rope` (lines 104-112). Causal masking is generated automatically via `make_causal_mask` (lines 88-90) to prevent attention to future tokens, with optional support for padding masks and sliding-window attention constraints.

## Quantization Support

Throughout the forward pass, the architecture supports optional activation quantization via `_aq` hooks and KV tensor quantization via `maybe_quant_kv` (lines 22-26). These mechanisms enable 8-bit inference without modifying the model topology, reducing memory footprint for deployment.

## Running the Simple Attention Network

The following example demonstrates instantiating the SAN, processing tokens, and utilizing auxiliary heads:

```python

# 1️⃣ Build the model -------------------------------------------------

import jax
import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    mhc_lanes=4,
    attn_dim=0,          # 0 → use d_model

    flash=True,
    dtype="bfloat16",
)

model = SimpleAttentionNetwork(cfg)

# 2️⃣ Tokenise a dummy input -----------------------------------------

from needle.model.tokenizer import SimpleTokenizer
tok = SimpleTokenizer()
tokens = jnp.array([tok.encode("Hello world!")])   # shape (1, N)

# 3️⃣ Forward pass (logits only) --------------------------------------

variables = model.init(jax.random.PRNGKey(0), tokens)
logits = model.apply(variables, tokens)                # shape (1, N, vocab)

# 4️⃣ Retrieve hidden states (for probing) ---------------------------

hidden = model.apply(
    variables, 
    tokens, 
    method=SimpleAttentionNetwork.hidden_states
)

# hidden shape: (num_layers, batch, seq_len, d_model)

# 5️⃣ Contrastive embedding (useful for retrieval) -------------------

query_emb, _, log_temp = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_contrastive,
    query_tokens=tokens,
    tool_tokens=tokens,
)

# 6️⃣ Confidence score ------------------------------------------------

conf_score = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_confidence,
)
print("Confidence:", conf_score)

```

The `model.init` and `model.apply` methods follow standard Flax patterns. Optional arguments like `quant`, `return_mtp`, and `window` enable 8-bit inference, multi-token prediction outputs, and sliding-window attention respectively.

## Summary

- The **Simple Attention Network** eliminates dense feed-forward parameters by using **Hadamard-MLP** layers with orthogonal Walsh-Hadamard transforms and element-wise gates.
- **Engram memory** layers provide external KV caches that inject n-gram statistics into specific attention layers via hash-based lookups.
- **Grouped-Query Attention (GQA)** reduces KV cache memory bandwidth by sharing key/value heads across query heads.
- **Multi-Lane Hyper-Connections** process an auxiliary pathway with next-token embeddings to enhance prediction quality.
- **Auxiliary heads** (Contrastive and Confidence) enable retrieval-based training and uncertainty quantification without external models.
- Full implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), with tokenization handled by [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py).

## Frequently Asked Questions

### What distinguishes the Simple Attention Network from standard transformer architectures?

The SAN replaces the conventional dense feed-forward network with a **Hadamard-MLP** that uses orthogonal transforms rather than learnable weight matrices, and it augments attention with **engram memory** layers that provide external KV caches populated by n-gram statistics. According to the cactus-compute/needle source code, these modifications reduce parameter counts while maintaining representational capacity through structured transforms and external memory lookups.

### How does engram memory work in Needle 2?

Engrams are implemented as fast key-value tables instantiated for layers specified in `config.engram_layers` (lines 90-95 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)). They use hash-based indexing (`engram_indices`) to store and retrieve n-gram statistics, effectively giving the model access to multi-token pattern memory without increasing the trainable parameters of the attention mechanism itself.

### What are the Multi-Lane Hyper-Connections (MHC) used for?

The MHC pathway (defined by `mtp_*` members in lines 96-104) creates a secondary processing lane that concatenates hidden states with next-token embeddings, mixes them through a learnable dense layer (`mtp_combine`), and processes them through an additional transformer block. This architecture enriches the representation for next-token prediction by explicitly modeling future context in a separate computational stream before final normalization.

### How can I extract confidence scores or embeddings from the model?

The SAN provides two dedicated methods: `SimpleAttentionNetwork.forward_confidence` returns a scalar confidence score via the `ConfidenceHead`, while `SimpleAttentionNetwork.forward_contrastive` produces L2-normalized embeddings suitable for retrieval tasks via the `ContrastiveHead`. Both methods can be invoked through Flax's `model.apply` by specifying the `method` parameter, as shown in the implementation example above.