How Needle's Engram Key-Value Memory Mechanism Works for Efficient Retrieval

Needle's engram KV memory stores compressed token histories in learned hash tables that enable O(1) retrieval of long-range context without expanding the standard attention window.

The cactus-compute/needle repository implements a novel transformer augmentation that addresses the fundamental scaling limitation of self-attention: quadratic complexity with sequence length. By introducing an engram key-value memory mechanism, Needle provides efficient access to distant context through fixed-size learned tables rather than full pairwise attention.

This article breaks down the complete engram KV pipeline—from configuration through retrieval—using the actual source implementation.

---

Core Components of the Engram System

The engram mechanism consists of four tightly coupled components defined across needle/model/architecture.py and needle/model/decode.py:

Engram Geometry

The engram_geometry(cfg) function (lines 20-27 in needle/model/architecture.py) computes the dimensional layout for all engram tables:

• orders: tuple of n-gram orders (e.g., (2, 3) for bigrams and trigrams)
• heads: number of attention heads (user-specified or derived from d_model)
• sub_dim: per-table dimension, calculated as d_model // (len(orders) * heads)

This geometry determines how many tables exist and how memory is partitioned among them.

Engram Indices

The engram_indices(tokens, orders, heads, slots) function (lines 46-58) provides deterministic hashing. It uses a 32-bit linear congruential generator with fixed constants _ENGRAM_SEED and _ENGRAM_PRIME to map token sequences to slot indices.

The hash guarantees that identical token sequences always resolve to the same table position—a critical property for stable retrieval.

Engram Module

The Engram class (lines 81-100) encapsulates the learnable tables:

• embedding: the actual KV storage ([num_orders, heads, slots, sub_dim])
• key_proj and value_proj: dense layers that project retrieved embeddings
• taps: learned convolution weights for temporal aggregation

Runtime KV Lookup

The _engram_kv helper in needle/model/decode.py (lines 56-84) performs the actual retrieval, embedding projection, and tap convolution during inference.

---

Step-by-Step: How Engram Retrieval Works

1. Configuration via TransformerConfig

Users define engram behavior through TransformerConfig:

from needle.model.architecture import TransformerConfig, engram_geometry

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    engram_orders=(2, 3),      # use 2-gram and 3-gram tables

    engram_heads=0,             # auto-derive heads from d_model

    engram_slots=8192,          # fixed table size

    engram_layers=(2, 15),      # only apply at layers 2 and 15

)

These parameters control memory capacity, retrieval granularity, and computational overhead.

2. Geometry Computation

orders, heads, sub_dim = engram_geometry(cfg)
print(f"orders: {orders}, heads: {heads}, sub-dim: {sub_dim}")

# Output: orders: (2, 3), heads: 12, sub-dim: 32

The sub_dim of 32 means each of 24 total tables (2 orders × 12 heads) contributes a 32-dimensional vector, summing to the full 768-dimensional model.

3. Index Generation for Token Sequences

import jax.numpy as jnp
from needle.model.architecture import engram_indices

tokens = jnp.array([[5, 12, 23, 7, 9]])  # batch_size=1, sequence_length=5

indices = engram_indices(tokens, orders=(2, 3), heads=12, slots=8192)
print(indices.shape)  # (1, 5, 24) — 24 indices per position (2×12)

The function slides over the sequence, computing for each position the hash of the local n-gram context.

4. Table Lookup and Projection

Within Engram.__call__ (lines 92-99):

• Retrieve embeddings at computed indices: embedding[order, head, slot]
• Mask invalid n-grams using ngram_ok boolean mask
• Flatten and project through key_proj and value_proj dense layers

5. Tap Convolution for Temporal Context

The _engram_kv function applies ENGRAM_CONV_TAPS shifted convolutions:

# Simplified conceptual flow from decode.py lines 67-78

for tap_idx in range(ENGRAM_CONV_TAPS):
    shifted = _shift_right(values, tap_idx)  # temporal offset

    weighted = shifted * taps[tap_idx]        # learned weight

    output += weighted

This lightweight convolution captures sequential patterns without recurrent computation.

6. Integration with Main Attention

In _forward_cached (needle/model/decode.py), engram KV tensors blend with standard attention only at configured layers:

• A gating scalar alpha controls mixture proportions
• Standard attention handles local relationships
• Engram KV memory provides efficient long-range retrieval at fixed cost

---

Why Engram KV Memory Achieves O(1) Efficiency

Traditional attention requires computing and attending to the full KV cache, growing linearly in memory and quadratically in computation with sequence length.

Needle's engram mechanism breaks this dependency:

Aspect	Standard Transformer	Needle with Engrams
Long-range context storage	Full KV cache	Fixed-size learned tables
Retrieval complexity	O(sequence length)	O(1) per token
Memory scaling	Linear with length	Constant (configurable)
Attention window	Explicitly limited	Effectively unbounded

The key insight: by compressing history into hash-addressable tables, Needle trades exact memorization for learned associative retrieval. The model learns which historical patterns to preserve and how to reconstruct useful context from compact representations.

---

Complete Inference Example

from needle.model.decode import generate_cached
from needle.model.run import init_params
from needle.model.tokenizer import Tokenizer

# Initialize from config defined above

params = init_params(cfg)  # or load trained checkpoint

tokenizer = Tokenizer()

# Generate with engram KV active

output = generate_cached(
    config=cfg,
    params=params,
    tokenizer=tokenizer,
    prompt="The quick brown fox",
    max_new_tokens=20,
)
print(output)

During generation, the engram tables continuously index recent context, allowing the model to retrieve patterns from arbitrarily distant prior tokens without expanding the causal mask.

---

Summary

• Engram geometry in architecture.py partitions model dimension across hash tables based on configured orders and heads.
• Deterministic hashing via engram_indices() maps token sequences to fixed table slots in O(1) time.
• Learned projections (key_proj, value_proj) and tap convolutions reconstruct useful KV representations from compressed table entries.
• Layer-selective integration via engram_layers and gating scalar alpha balances local attention with global retrieval.
• Constant memory footprint regardless of sequence length makes the engram key-value memory mechanism suitable for extremely long contexts.

---

Frequently Asked Questions

How does Needle's engram hashing avoid collisions?

The engram_indices function uses a 32-bit linear congruential generator with prime-derived mixing (_ENGRAM_PRIME = 2654435761). While collisions are statistically inevitable with finite slots, the learned nature of the tables allows the model to tolerate or even exploit collision patterns during training. The system prioritizes fast, deterministic indexing over perfect uniqueness.

Can engram orders be changed after training?

No. The engram_orders tuple determines table structure at initialization and becomes baked into parameter shapes. Changing orders requires retraining because the embedding tensor dimensions ([num_orders, heads, slots, sub_dim]) and all downstream projections depend on this configuration.

What determines the optimal number of engram slots?

The engram_slots parameter represents a classic capacity-accuracy tradeoff. More slots reduce collisions and improve retrieval fidelity but increase memory and parameter count. The default 8192 slots balance these factors for typical language modeling scales, but task-specific tuning may help.

How does tap convolution differ from standard attention?

Tap convolution in _engram_kv applies fixed-offset, learned-weight blending of value projections across ENGRAM_CONV_TAPS positions. Unlike attention's content-dependent weighting, taps use predetermined temporal offsets with trainable scalar weights—substantially cheaper while still capturing local sequential structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}