How the Engram Key-Value Memory Works in Needle 2 Architecture

The Engram Key-Value Memory in Needle 2 replaces the linearly-growing Transformer KV cache with a fixed-size, learned hash table that retrieves long-range context via deterministic n-gram hashing, enabling arbitrarily long sequences with bounded memory.

Needle 2, maintained in the cactus-compute/needle repository, implements a sparse, high-capacity memory system called the Engram that stores attention keys and values outside the traditional token-by-token cache. Unlike standard Transformers where memory scales with sequence length, the Engram Key-Value Memory uses learned embeddings indexed by hashed n-grams to provide constant-time access to semantic context from any position in the input.

Geometry and Configuration of the Engram

The memory layout is defined by three hyper-parameters stored in the model configuration:

• engram_orders – tuple defining n-gram orders (e.g., (2, 3))
• engram_heads – number of heads per order (defaults to ⌈d_model / (len(orders) × sub_dim)⌉)
• engram_slots – total hash table size (defaults to 8192)

Computing Memory Dimensions

The helper function engram_geometry in needle/model/architecture.py calculates the final geometry used to initialize embedding tables:

# Lines 20-25 in needle/model/architecture.py

def engram_geometry(orders, d_model, sub_dim):
    """Compute (orders, heads_per_order, sub_dim) for Engram tables."""
    num_orders = len(orders)
    heads = math.ceil(d_model / (num_orders * sub_dim))
    return orders, heads, sub_dim

This geometry determines the shape of the learned memory banks that will store the key-value pairs.

Token-to-Slot Mapping via Deterministic Hashing

For every input sequence, Needle 2 computes hash indices that map token windows to specific slots in the Engram tables. This process is handled by the engram_indices function.

The engram_indices Implementation

Located at lines 46-58 in needle/model/architecture.py, this function builds a deterministic hash for each (order, head) pair:

indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

The implementation uses a multiply-add hash combining _ENGRAM_SEED and _ENGRAM_PRIME constants. It mixes token IDs with shifted versions of the token stream (up to the order length) and reduces the result modulo slots, producing an integer index per token per head. This deterministic approach ensures that identical n-grams always map to the same memory location across different forward passes.

Embedding Table Look-up and KV Projection

Each Engram module owns a learned embedding table shaped (num_tables, slots, sub_dim), initialized in the module's constructor at lines 81-88 of needle/model/architecture.py.

Memory Retrieval and Masking

During the forward pass, the model retrieves vectors using the pre-computed indices:

tables = self.param("embedding", default_init(),
                    (self.num_tables, self.slots, self.sub_dim))
fetched = tables[jnp.arange(self.num_tables), indices]

# Shape: (B, T, num_tables, sub_dim)

The retrieved vectors pass through two binary masks before becoming valid keys and values:

• ngram_ok – Enforced in _engram_kv (lines 14-16), this mask zeros out entries that would cross token-order boundaries, ensuring the n-gram window is respected.
• tap_ok – Implemented in Engram.__call__ (lines 102-106), this enables a small convolutional "tap" filter defined by ENGRAM_CONV_TAPS that mixes recent values spatially.

Projection and Quantization

After masking and reshaping (e = fetched.reshape(*indices.shape[:2], self.num_tables * self.sub_dim)), the vectors are linearly projected to the model dimension:

k = nn.Dense(self.d_model)(e)   # Key projection

v = nn.Dense(self.d_model)(e)   # Value projection

Both projections support optional quantization via the _aq parameter for mixed-precision inference, as seen in Engram.__call__ at lines 95-98.

Integration into the Transformer Stack

The SimpleAttentionNetwork builds Engram modules for layers specified in the configuration (engram_layers), instantiating them in __init__ at lines 90-95.

KV Retrieval Pipeline

The internal _engram_kv method (lines 13-19) executes the retrieval workflow:

1. Computes hash indices via engram_indices.
2. Builds ngram_ok and tap_ok masks.
3. Invokes each Engram module to obtain (k, v) tuples.
4. Stacks results into tensors shaped (num_engram_layers, B, T, d_model).

Gated Merge in Transformer Blocks

Inside each Block, the Engram KV pairs merge with token representations via a learned gating mechanism implemented in Block.__call__ at lines 19-25:

alpha = nn.sigmoid(
    jnp.einsum("btd,sbtd->sbt", _rms_unit(x), _rms_unit(ek))
    / math.sqrt(self.d_model)
)
x = x + jnp.einsum("s,sbt,sbtd->btd", site_flags, alpha, ev)

This sigmoid-controlled gate allows each layer to dynamically weight the outside-the-window context retrieved from the Engram memory against the local token representation.

Memory Efficiency and Long-Range Context

The Engram Key-Value Memory offers distinct architectural advantages over traditional caching:

Memory size – Traditional caches grow linearly with sequence length (tokens × heads), while the Engram maintains a fixed footprint of slots × sub_dim regardless of context length.

Long-range recall – Standard KV caches are limited to recent context unless the full history is retained. The Engram retrieves from any past position via hash collision, enabling arbitrarily long context windows.

Computational cost – Instead of O(N²) attention over growing caches, the Engram requires O(1) constant-time lookups plus cheap linear projections.

Sparsity – Unlike dense traditional caches that store all tokens, the Engram learns sparse access patterns where only hashed slots contain information.

Learnability – While standard KV caches derive directly from token embeddings, Engram tables are fully learnable with convolutional taps, allowing the model to store compressed semantic memory rather than literal token representations.

End-to-End Implementation Example

The following example demonstrates configuring and running inference with the Engram Key-Value Memory enabled:

import jax.numpy as jnp
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# Configure model with Engram layers active at positions 2-15

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),          # Tuple specifying active layer range

    engram_orders=(2, 3),
    engram_slots=8192,
)

model = SimpleAttentionNetwork(cfg)

# Dummy input: batch_size=1, seq_len=10

tokens = jnp.arange(10)[None, :]

# Forward pass automatically uses Engram KV memory

logits = model(tokens)

# Inspect the generated Engram KV tensors

engram_kv = model._engram_kv(tokens, mask=None, quant=False)
keys, values = engram_kv
print(keys.shape, values.shape)   # (num_engram_layers, 1, 10, d_model)

In this implementation, needle/model/architecture.py transparently handles the Engram initialization, hashing, and integration, while SimpleAttentionNetwork orchestrates the retrieval pipeline defined in _engram_kv.

Summary

• Fixed-size storage – The Engram uses hash tables with configurable slots (default 8192) independent of sequence length, solving the linear memory growth problem of standard Transformers.
• Deterministic hashing – The engram_indices function maps token n-grams to slots using seeded multiply-add hashing in needle/model/architecture.py.
• Learned embeddings – Each Engram module maintains learnable tables shaped (num_tables, slots, sub_dim) with optional convolutional taps for local context mixing.
• Gated integration – Transformer blocks merge Engram KV pairs via sigmoid gating in Block.__call__, allowing dynamic weighting of long-range context.
• Quantization support – The architecture supports mixed-precision training and inference through optional _aq quantization on projection layers.

Frequently Asked Questions

How does the Engram Key-Value Memory differ from standard Transformer KV caching?

Traditional KV caches store every token's key and value vectors in a linear buffer that grows with sequence length, consuming O(N) memory. The Engram Key-Value Memory compresses context into fixed-size hash tables accessed by n-gram hashes, maintaining O(1) memory usage regardless of input length while enabling retrieval from arbitrarily distant positions.

What prevents hash collisions from corrupting the Engram memory?

The engram_indices function uses deterministic hashing modulo engram_slots, meaning collisions are intentional and frequency-based. The learned embedding tables adapt during training to store superposed representations in collision slots, effectively learning to compress multiple n-gram contexts into shared memory locations through gradient descent, similar to how standard embedding layers handle vocabulary hashing.

Which transformer layers utilize the Engram memory?

The engram_layers parameter in TransformerConfig specifies a tuple (start, end) defining which layer indices receive Engram KV pairs. According to the SimpleAttentionNetwork initialization in needle/model/architecture.py, layers outside this range operate with standard self-attention only, allowing architectures to mix local attention (early/late layers) with Engram-augmented attention (middle layers).

Is the Engram Key-Value Memory compatible with quantization?

Yes, the Engram implementation in needle/model/architecture.py supports mixed-precision inference. The Engram.__call__ method accepts a quant parameter that triggers _aq (activation quantization) on the dense projections that convert retrieved embeddings into keys and values, reducing memory bandwidth during retrieval without altering the hash table storage format.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}