# Simple Attention Network (SAN) Architecture in Needle: A Technical Deep Dive

> Explore the Simple Attention Network SAN architecture in Needle. Learn how this Transformer-style encoder-decoder uses Engram memory and Hadamard MLPs to enhance attention for advanced machine learning tasks.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-11

---

**The Simple Attention Network (SAN) is a Transformer-style encoder-decoder that augments standard multi-head attention with a learned Engram memory, Hadamard MLPs, and auxiliary heads for contrastive learning and confidence estimation, implemented in the `SimpleAttentionNetwork` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).**

The **Simple Attention Network (SAN)** serves as the core neural architecture powering Needle, an open-source inference framework maintained by cactus-compute. Unlike standard Transformer implementations, SAN integrates associative memory mechanisms and task-specific prediction heads directly into its forward pass. This guide examines the complete SAN implementation by tracing its components from embedding initialization to autoregressive generation.

## Core Architecture Components

### Token Embedding and Scaling

During initialization in `setup()`, SAN creates an embedding matrix using **Flax's `nn.Embed`** configured with a standard deviation of 0.02. The implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (around line 83) scales token embeddings by the square root of the model dimension to preserve variance across depth:

```python
self.embedding = nn.Embed(cfg.vocab_size, cfg.d_model, embedding_init=jinit.normal(stddev=0.02))
self.embed_scale = math.sqrt(cfg.d_model)

```

Input tokens are embedded and multiplied by `embed_scale` before entering the main transformer stack.

### The Stack and Transformer Blocks

The **Stack** module, defined at line 76 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py), encapsulates the full sequence of transformer layers. Each layer is a **Block** containing:

- **Multi-Head Attention** (`MultiHeadAttention`) with optional flash-attention support for GPU acceleration
- **Hadamard MLP** (`HadamardMLP`) replacing standard feed-forward networks with Walsh-Hadamard transforms for computational efficiency
- **Layer-wise gating** to regulate residual contributions

The stack leverages `nn.scan` or `nn.remat` (when `remat=True`) to enable efficient JIT compilation and reversible memory checkpointing.

### Engram Memory System

SAN optionally incorporates **Engram memory**, a learned associative memory mechanism defined at line 81 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py). The `_engram_kv` method computes hashed n-gram indices and retrieves key/value tables that fuse with standard attention KV caches. This design provides sub-linear scaling with sequence length by augmenting the context with hashed lookups rather than full attention over all prior tokens.

### Auxiliary Prediction Heads

Beyond the main language modeling head, SAN computes three auxiliary outputs:

- **ContrastiveHead** (line 44): Projects hidden states to a low-dimensional space (`cfg.contrastive_dim`) with learned temperature parameters for retrieval and ranking tasks
- **ConfidenceHead** (line 64): Predicts scalar confidence scores for generated sequences using probe pooling mechanisms
- **MTP (Multi-Task Prediction) block** (lines 99-101): An auxiliary pathway that fuses final hidden states with next-step token embeddings to produce secondary logits for tool-calling or structured output generation

## Configuration and Initialization

SAN receives hyperparameters through the **`TransformerConfig`** dataclass defined at line 58 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py). Key configuration fields include:

- `d_model`: Hidden dimension size
- `num_heads` / `num_kv_heads`: Attention head configuration (supports GQA)
- `num_layers`: Depth of the transformer stack
- `vocab_size`: Token vocabulary dimensions
- `max_seq_len`: Maximum context window
- `engram_layers`, `engram_orders`, `engram_heads`, `engram_slots`: Controls for the Engram memory system

The `quant` flag enables fake quantization throughout the model for training-time approximation of low-bit inference, utilizing helpers from [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) such as `fake_quant_act` and `maybe_quant_kv`.

## The Forward Pass Execution Flow

The `__call__` method of `SimpleAttentionNetwork` executes the following sequence:

1. **Mask Creation**: Generates a causal attention mask if none is provided
2. **Embedding**: Scales input token embeddings by `sqrt(d_model)`
3. **Positional Encoding**: Pre-computes **Rotary Positional Embeddings (RoPE)** via `precompute_rope_freqs` (line 97)
4. **Engram Retrieval**: Optionally computes Engram KV pairs through `_engram_kv`
5. **Stack Processing**: Passes embeddings through the transformer stack with attention and MLP blocks
6. **Logit Projection**: Projects final hidden states back to vocabulary space using the transposed embedding matrix
7. **MTP Pathway**: If `return_mtp=True`, computes secondary logits through the MTP block

All operations respect the quantization configuration when `quant` is enabled.

## Generating Text with SAN: Practical Example

The following example demonstrates loading a checkpoint, instantiating SAN, and generating text. This mirrors the CLI implementation in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py):

```python
import pathlib
from needle.model.run import load_checkpoint, generate
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig
from needle.model.tokenizer import get_tokenizer

# 1️⃣ Load a checkpoint (downloaded automatically if missing)

ckpt_path = pathlib.Path("checkpoints/needle_v1.ckpt")
params, cfg = load_checkpoint(ckpt_path)

# 2️⃣ Instantiate the model

model = SimpleAttentionNetwork(cfg)

# 3️⃣ Retrieve the tokenizer (downloads from HF if needed)

tokenizer = get_tokenizer(cfg.vocab_size)

# 4️⃣ Generate a continuation

prompt = "The most surprising thing about"
output = generate(
    model=model,
    params=params,
    tokenizer=tokenizer,
    prompt=prompt,
    max_new_tokens=64,
    temperature=0.0,      # deterministic sampling

    seed=42,
    stream=False,        # returns the full text at once

)

print("🖋️", output)

```

**Key functions referenced:**
- `load_checkpoint`: Loads serialized parameters from line 62 of [`run.py`](https://github.com/cactus-compute/needle/blob/main/run.py)
- `get_tokenizer`: Returns a SentencePiece tokenizer from line 35 of [`tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/tokenizer.py)
- `generate`: Autoregressive sampling loop defined at line 77 of [`run.py`](https://github.com/cactus-compute/needle/blob/main/run.py)

### Advanced Capabilities

SAN exposes several specialized methods for downstream tasks:

- **Contrastive encoding**: `model.encode_contrastive(tokens)` returns normalized representations for semantic search
- **Confidence scoring**: `model.forward_confidence(tokens)` yields per-sequence uncertainty estimates
- **Hidden state extraction**: `model.hidden_cells(tokens, window=128)` returns per-layer activations for analysis or fine-tuning workflows

## Summary

- **Simple Attention Network (SAN)** combines standard Transformer blocks with novel components including Engram memory and Hadamard MLPs in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)
- **Configuration** is managed through the `TransformerConfig` dataclass (line 58), controlling model dimensions, attention heads, and memory parameters
- **Engram memory** provides hashed associative lookup capabilities that augment standard KV caches, scaling efficiently with sequence length
- **Auxiliary heads** enable contrastive learning, confidence calibration, and multi-task prediction without requiring separate model instances
- **Quantization support** is built into the forward pass, allowing training-time simulation of low-bit inference using utilities from [`quantize.py`](https://github.com/cactus-compute/needle/blob/main/quantize.py)

## Frequently Asked Questions

### What distinguishes SAN from standard Transformer architectures?

SAN extends the traditional Transformer encoder-decoder by integrating an **Engram memory** system for hashed n-gram lookups and replacing standard feed-forward layers with **Hadamard MLPs** for computational efficiency. These modifications, located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), enable better scaling properties and auxiliary task capabilities while maintaining compatibility with standard attention mechanisms.

### How does the Engram memory system work?

The Engram system computes hashed indices for input sequences via the `_engram_kv` method and retrieves learned key/value pairs that **fuse with standard attention KV caches**. This associative memory mechanism allows the model to access relevant context through hash-based lookups rather than full attention computation, reducing memory complexity as sequence length increases.

### What are the auxiliary heads used for in SAN?

SAN includes three specialized heads: the **ContrastiveHead** projects representations for retrieval and ranking tasks, the **ConfidenceHead** estimates prediction uncertainty, and the **MTP block** generates secondary logits for tool-calling or structured outputs. These heads share probe pooling mechanisms and operate on the final hidden states from the main transformer stack.

### How do I configure and instantiate a SAN model?

Create a `TransformerConfig` dataclass instance specifying `d_model`, `num_heads`, `num_layers`, and Engram parameters, then pass it to `SimpleAttentionNetwork(cfg)` as shown in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) line 78. Load parameters using `load_checkpoint()` from [`run.py`](https://github.com/cactus-compute/needle/blob/main/run.py) (line 62) and generate text via the `generate()` function with appropriate tokenizer configuration.