Engram KV Memory in Needle 2's Hadamard MLP: How the Hash-Based Key-Value Cache Works

Engram KV Memory is a learned, hash-based key-value cache that retrieves n-gram contextual embeddings and injects them into Needle 2's attention mechanism, enabling the Hadamard MLP to operate on representations enriched with long-range, n-gram-aware information.

Needle 2 from the cactus-compute/needle repository introduces a novel memory mechanism that augments transformer architectures without the computational cost of standard KV caching. This article explains what Engram KV Memory is, how it constructs hashed indices from token sequences, and how its outputs flow into the Hadamard MLP to improve representation quality.

---

What Is Engram KV Memory?

Engram is Needle 2's implementation of KV-memory—a learned embedding storage system addressed by hashed n-gram indices rather than direct token positions. Unlike conventional key-value caches that grow with sequence length, Engram uses a fixed-size slot table with hash-based addressing, making memory consumption predictable regardless of input length.

The core idea: instead of storing per-position keys and values, the model learns to associate n-gram patterns (contiguous token sequences) with embedding vectors. When a particular n-gram appears in the input, the model retrieves its associated representation and projects it into keys and values for attention.

---

How Engram Constructs Indices and Retrieves Embeddings

The Engram mechanism operates through four sequential stages inside architecture.py.

Hashing Tokens into N-Gram Indices

The engram_indices function transforms raw tokens into hash positions using configurable n-gram orders and per-head randomization. This happens in architecture.py lines 46-57:

indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

The hashing combines:

• Orders: Which n-gram sizes to use (e.g., 2-grams, 3-grams)
• Seeds: Per-head randomization to reduce hash collisions
• Slot count: Fixed table size (e.g., 8192 slots)

Embedding Table Lookup with Masking

Each Engram layer maintains learnable embedding tables of shape (num_tables, slots, sub_dim). The lookup applies two critical masks:

tables = self.param("embedding", default_init(),
                    (self.num_tables, self.slots, self.sub_dim))       # [architecture.py#L91-L94]

fetched = tables[jnp.arange(self.num_tables), indices]               # [architecture.py#L94-L95]

• ngram_ok: Disables entries at positions without sufficient context for a full n-gram
• tap_ok: Excludes positions outside the causal window

Projection to Keys and Values

The retrieved embeddings are reshaped and projected into full-dimensional keys and values:

k = nn.Dense(self.d_model, …)(e)                                   # [architecture.py#L98-L99]

v = nn.Dense(self.d_model, …)(e)                                   # [architecture.py#L100-L101]

The values then undergo a tap convolution—a 1-D convolution that mixes recent values across time steps:

v = sum(taps[j] * _shift_right(v, j * self.conv_dilation) … )       # [architecture.py#L102-L105]

This tap mechanism gives Engram a limited short-term memory effect, complementing the long-range n-gram associations.

---

Integrating Engram KV Memory into the Transformer Stack

The SimpleAttentionNetwork class orchestrates how Engram outputs reach the model's processing layers.

Engram KV Creation in SimpleAttentionNetwork

The method _engram_kv aggregates keys and values from all configured Engram instances:

engram_kv = self._engram_kv(tokens, mask, quant)                    # [architecture.py#L124-L129]

This tuple (k, v) is passed through the transformer stack:

x, _ = self.stack(x, …, engram_kv=engram_kv, …)                     # [architecture.py#L125-L127]

Block-Level Injection

Inside each Block, the attention mechanism receives these enriched keys and values augmenting its standard computation:

x = MultiHeadAttention(...)(x, …)                                   # [architecture.py#L30-L33]

The attention output—now incorporating n-gram-aware information from Engram—proceeds to the Hadamard MLP.

---

How Engram Memory Feeds the Hadamard MLP

The HadamardMLP is where the memory-augmented representations get processed. This module implements a fast feed-forward layer using Hadamard transformations for computational efficiency.

Processing Order in Each Block

The forward pass structure in Block.__call__ (lines 30-38) shows the critical sequence:

x = MultiHeadAttention(...)(x, …)                                   # [architecture.py#L30-L33]

x = HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)   # [architecture.py#L35-L37]

The Engram KV Memory enriches attention outputs before they reach the Hadamard MLP. This means:

1. The Hadamard MLP receives activations informed by n-gram patterns from the entire sequence history
2. The orthogonal transformation in the MLP operates on semantically richer representations
3. Computational efficiency is preserved—Hadamard transforms are faster than dense matrix multiplications

Why This Architecture Matters

The combination achieves two goals typically in tension:

• Expressive memory: N-gram hashing captures meaningful linguistic patterns without attending to every past position
• Computational efficiency: Fixed-size Engram tables and fast Hadamard transforms keep inference costs controlled

---

Practical Examples: Working with Engram KV Memory

Instantiating a Model with Engram Enabled

import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(
    d_model=512,
    num_heads=8,
    num_kv_heads=4,
    num_layers=12,
    engram_layers=(2, 9),          # layers receiving Engram KV memory

    engram_orders=(2, 3),          # 2-gram and 3-gram hashing

    engram_slots=8192,
)

model = SimpleAttentionNetwork(cfg)
tokens = jnp.ones((1, 16), dtype=jnp.int32)
logits = model(tokens)            # Engram KV memory queried automatically

Directly Querying Engram KV Memory

import jax.numpy as jnp
from needle.model.architecture import SimpleAttentionNetwork, engram_indices, _mask_diag

model = SimpleAttentionNetwork(cfg)
tokens = jnp.arange(1, 9)[None, :]
mask = model.make_causal_mask(tokens.shape[1])

orders, heads, _ = model.engram_geometry(cfg)
indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

ngram_ok = jnp.stack([_mask_diag(mask, o - 1) for o in orders for _ in range(heads)], axis=-1)
tap_ok = jnp.stack([_mask_diag(mask, j * max(orders)) for j in range(4)], axis=0)

engram = model.engrams[0]
k, v = engram(indices, ngram_ok, tap_ok, quant=False)
print("Key shape:", k.shape)      # (batch, seq_len, d_model)

print("Value shape:", v.shape)    # (batch, seq_len, d_model)

Inspecting Hadamard MLP Inputs

def forward_and_collect(x):
    final, hidden = model.stack(
        x, mask=mask, rope=model._rope(x.shape[1]),
        engram_kv=model._engram_kv(tokens, mask, False)
    )
    return hidden

pre_hada = forward_and_collect(model.embedding(tokens) * model.embed_scale)
print(pre_hada.shape)   # (num_layers, batch, seq_len, d_model)

---

Key Implementation Files

File	Purpose
needle/model/architecture.py	Engram class definition, hashing utilities (engram_indices, _mask_diag), transformer integration
architecture.py#L81-L107	Engram KV storage and projection to keys/values
architecture.py#L124-L129	Engram KV pair creation and stack injection
architecture.py#L30-L38	Block attention and HadamardMLP sequencing
architecture.py#L86-L95	Engram instance construction in SimpleAttentionNetwork

---

Summary

• Engram KV Memory uses hash-based n-gram indexing to retrieve learned contextual embeddings from fixed-size tables
• The mechanism produces keys and values through dense projection and tap convolution, then injects them into multi-head attention
• Hadamard MLP processes attention outputs that already contain n-gram-aware information, combining memory richness with computational efficiency
• Configuration via TransformerConfig controls which layers receive Engram memory, which n-gram orders to hash, and table size

---

Frequently Asked Questions

How does Engram KV Memory differ from standard transformer KV caching?

Standard KV caches store per-position keys and values that grow linearly with sequence length. Engram uses a fixed-size hash table addressed by n-gram patterns, making memory consumption constant regardless of sequence length. According to the cactus-compute/needle source code, this is implemented through the engram_indices function that hashes token n-grams into bounded slot indices rather than maintaining position-based storage.

What n-gram orders should I use with Engram?

The engram_orders configuration parameter accepts a tuple of integers specifying which n-gram sizes to hash. Common choices are (2, 3) for bigrams and trigrams, or (2, 3, 4) for longer contexts. Larger orders capture more specific patterns but increase hash collision risk with fixed slot counts; the default engram_slots=8192 balances capacity and memory usage.

Can I disable Engram memory in Needle 2?

Yes—set engram_layers=() in your TransformerConfig to disable all Engram memory. The model will fall back to standard attention without n-gram augmentation. The SimpleAttentionNetwork constructor in architecture.py lines 86-95 conditionally creates Engram instances only when layers are specified.

Why combine Engram with Hadamard transformations specifically?

The Hadamard MLP provides computationally efficient feed-forward processing through fast orthogonal transforms. By placing Engram-augmented attention before this efficient MLP, Needle 2 achieves rich contextual representations without the quadratic costs of full attention expansion or large dense MLPs. As implemented in architecture.py lines 30-38, this ordering ensures the Hadamard transform operates on the most informative possible activations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}