# How the Simple Attention Network Architecture Works in Needle 2

> Discover how the Simple Attention Network architecture in Needle 2 boosts inference performance with Hadamard-MLPs, engram memory, and Grouped-Query Attention, all while reducing computational cost.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-14

---

**The Simple Attention Network (SAN) in Needle 2 replaces traditional transformer feed-forward networks with Hadamard-MLPs, integrates hash-based engram memory, and uses Grouped-Query Attention to deliver high-performance inference with a reduced computational footprint.**

The `cactus-compute/needle` repository implements Needle 2 as a compact yet powerful language model built around the Simple Attention Network architecture. This design eliminates conventional dense layers in favor of orthogonal transforms and adds external n-gram memory through engrams, creating a streamlined alternative to standard transformer stacks.

## Model Configuration and Embedding

### TransformerConfig Dataclass

All SAN hyperparameters are centralized in the `TransformerConfig` dataclass defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 58-71). This configuration specifies `d_model`, `num_heads`, `engram_layers`, and `mhc_lanes`, while computing derived values such as `attn_dim` and JAX dtype settings. The config drives every subsequent layer's dimensions and behavior, ensuring consistency across the embedding table and attention blocks.

### Embedding Layer Setup

During `SimpleAttentionNetwork.setup()`, the model initializes an `nn.Embed` table of shape `vocab_size × d_model` and applies the standard "sqrt-dim" scaling trick. Specifically, the embedding weights are multiplied by √`d_model` to stabilize early training dynamics (lines 83-85). This scaled embedding serves as the initial representation for input tokens before they enter the processing stack.

## Core Architectural Components

### Engram Memory System

For each layer specified in `config.engram_layers`, the SAN constructs an `Engram` instance (lines 90-95). These engrams function as fast key-value tables that store n-gram statistics using hash-based indexing via `engram_indices`. During the forward pass, engrams provide additional KV pairs to the attention mechanism, effectively augmenting the context window with cheap, external memory that requires minimal compute overhead compared to standard attention caches.

### Processing Stack with GQA and Hadamard-MLPs

The heavy lifting occurs within the `Stack` module (lines 85-86), which scans a `_ScanBody` containing `Block` layers across all transformer depths. Each `Block` contains three critical sub-components:

- **Multi-Head GQA Attention** (`MultiHeadAttention`): Implements Grouped-Query Attention with optional Flash-Attention support, reducing KV cache memory bandwidth by sharing key-value heads across query heads.
- **Hadamard-MLP**: A dense-free feed-forward network that applies an orthogonal Walsh-Hadamard transform (`_walsh_matrix`) followed by element-wise non-linearities, eliminating learnable weight matrices in the MLP.
- **Engram KV Injection**: When enabled, engram outputs are concatenated with standard attention KV pairs to enrich the attention context.

### Multi-Lane Hyper-Connections (MHC)

After the main stack, the model processes hidden states through a multi-token prediction (MTP) pathway (lines 96-104). This **Multi-Lane Hyper-Connections** mechanism concatenates the original hidden states with "next-token" embeddings, mixes them via a learnable dense layer (`mtp_combine`), processes them through an additional `Block` (`mtp_block`), and applies a final RMS-norm (`mtp_final_norm`). This pathway generates richer representations specifically optimized for next-token prediction tasks.

## Output Heads and Projections

### Logits Projection

The final hidden representation projects back onto the vocabulary through a weight-tied operation using the transpose of the embedding matrix: `logits = x @ self.embedding.embedding.T` (lines 27-30). The SAN optionally returns both standard logits and MTP logits simultaneously when `return_mtp` is enabled.

### Auxiliary Heads

Two lightweight prediction heads sit atop the hidden cell outputs (lines 44-70):

- **ContrastiveHead**: Pools cell vectors using learned probes and L2-normalizes them to produce retrieval-style embeddings suitable for contrastive learning tasks.
- **ConfidenceHead**: Applies similar pooling but outputs a scalar confidence score indicating model certainty for each token position.

## Positional Encoding and Optimization

### RoPE and Causal Masking

The SAN automatically generates causal attention masks via `make_causal_mask` (lines 88-90) and supports optional padding masks for batched inference. Rotary Positional Embeddings (RoPE) are pre-computed using `precompute_rope_freqs` and applied to query and key tensors before attention computation (lines 104-112), allowing the model to generalize to sequence lengths not seen during training.

### Quantization Hooks

Throughout the forward pass, the architecture supports optional activation quantization via `_aq` functions and KV cache quantization through `maybe_quant_kv` (lines 22-26). These hooks enable 8-bit inference without modifying the model topology or requiring separate quantized checkpoints, reducing memory footprint during deployment.

## Running Inference with the Simple Attention Network

The following example demonstrates how to instantiate the SAN, initialize parameters, and utilize the auxiliary heads using the Flax API:

```python
import jax
import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork
from needle.model.tokenizer import SimpleTokenizer

# 1️⃣ Configure and build the model

cfg = TransformerConfig(
    vocab_size=8192,
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),
    mhc_lanes=4,
    attn_dim=0,          # 0 → use d_model

    flash=True,
    dtype="bfloat16",
)
model = SimpleAttentionNetwork(cfg)

# 2️⃣ Tokenize input

tok = SimpleTokenizer()
tokens = jnp.array([tok.encode("Hello world!")])   # shape (1, N)

# 3️⃣ Initialize and run forward pass

variables = model.init(jax.random.PRNGKey(0), tokens)
logits = model.apply(variables, tokens)            # shape (1, N, vocab_size)

# 4️⃣ Extract hidden states for analysis

hidden = model.apply(
    variables, tokens, 
    method=SimpleAttentionNetwork.hidden_states
)  # shape: (num_layers, batch, seq_len, d_model)

# 5️⃣ Generate contrastive embeddings

query_emb, _, log_temp = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_contrastive,
    query_tokens=tokens,
    tool_tokens=tokens,
)

# 6️⃣ Obtain confidence scores

conf_score = model.apply(
    variables,
    tokens,
    method=SimpleAttentionNetwork.forward_confidence,
)
print("Confidence:", conf_score)

```

## Summary

- The **Simple Attention Network** in Needle 2 replaces standard dense feed-forward layers with **Hadamard-MLPs**, using orthogonal Walsh-Hadamard transforms to reduce parameters.
- **Engram memory** provides cheap external KV pairs through hash-based n-gram tables, augmenting attention at specific layers without proportional compute cost.
- **Grouped-Query Attention (GQA)** and **Flash-Attention** support minimize memory bandwidth during inference.
- **Multi-Lane Hyper-Connections** generate enhanced representations for next-token prediction through an auxiliary processing pathway.
- Built-in **quantization hooks** (`_aq`, `maybe_quant_kv`) enable 8-bit inference without architectural changes.

## Frequently Asked Questions

### What is the difference between Hadamard-MLP and standard transformer MLPs?

Standard transformers use two dense layers with non-linearities between them, requiring substantial parameter counts and memory bandwidth. The **Hadamard-MLP** in Needle 2 replaces these dense matrices with fixed orthogonal Walsh-Hadamard transforms (`_walsh_matrix`) followed by element-wise gates, eliminating learnable weights in the feed-forward path while maintaining expressive power through non-linear transformations.

### How does engram memory improve model performance?

**Engrams** store n-gram statistics as additional key-value pairs outside the standard attention cache. According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 90-95), these are constructed only for layers specified in `engram_layers` and use hash-based indexing to retrieve relevant context. This allows the model to access longer-range dependencies and repetitive patterns without increasing the attention window size or computational complexity of the core attention mechanism.

### What are the ContrastiveHead and ConfidenceHead used for?

These auxiliary heads operate on the final hidden cell outputs for downstream tasks beyond next-token prediction. The **ContrastiveHead** (lines 44-55) generates L2-normalized embeddings suitable for retrieval and contrastive learning, while the **ConfidenceHead** (lines 56-70) outputs scalar values representing model certainty. Both heads use learned probe pools to aggregate information across sequence positions.

### Can the Simple Attention Network run with reduced precision?

Yes. The architecture includes explicit quantization support through `maybe_quant_kv` for the KV cache and `_aq` hooks for activations (lines 22-26). These functions enable 8-bit quantization during both training and inference without requiring model topology changes or separate quantized model definitions, significantly reducing memory requirements for deployment.