Understanding Engram Key-Value Memory in Needle 2's Transformer Architecture

The Engram Key-Value (KV) Memory in Needle 2 is a dedicated neural cache that augments standard transformer attention by storing and retrieving contextual information at specific layers, enabling efficient access to long-range or auxiliary signals without modifying core attention mechanics.

Needle 2, an open-source transformer architecture developed by Cactus Compute, introduces a novel memory mechanism called Engram that operates as a plug-in key-value store at selected transformer layers. This article examines how the Engram KV memory works, its integration points in the codebase, and why it represents a significant architectural enhancement over traditional attention-only models.

What Is Engram Key-Value Memory?

At its core, Engram Key-Value Memory is a small neural table-lookup system that maintains learned embedding tables for key-value retrieval. Unlike standard transformer attention, which recomputes key-value projections from hidden states at every layer, Engram pre-computes and caches KV pairs that can be rapidly accessed during forward passes.

The system is defined in needle/model/architecture.py where the Engram class (lines 81-106) implements the core lookup mechanism:

• Embedding tables — learned parameters indexed by token hash
• Dense projections — transform embeddings into key (k) and value (v) tensors
• Convolutional taps — optional short-range temporal context mixed into each site

How Engram Geometry Is Configured

The shape of the Engram memory is determined by engram_geometry() in needle/model/architecture.py (lines 120-124). This helper derives four critical dimensions from the model configuration:

Dimension	Configuration Parameter	Purpose
Tables	engram_orders	Number of n-gram hash tables
Heads	engram_heads	Parallel attention heads for Engram lookup
Slots	engram_slots	Hash bucket capacity per table
Sub-dimension	auto-derived	Per-head feature size

These parameters control the memory capacity and retrieval granularity. Higher engram_orders enable capturing more complex token patterns, while engram_slots determines collision resistance in the hash-based indexing scheme.

Enabling Engram Sites in Model Configuration

import needle
from needle.model.architecture import ModelConfig

# Enable Engram memory at layers 2 and 15, with 64 slots each.

cfg = ModelConfig(
    d_model=1024,
    num_layers=24,
    engram_layers=(2, 15),
    engram_slots=64,
    engram_orders=(2, 3),        # n-gram orders

    engram_heads=0,              # auto-derived from d_model

)

model = needle.model.Model(cfg)   # Internally creates Engram tables

The engram_layers tuple specifies which transformer layers receive the Engram KV injection. This selective placement allows the model to benefit from external memory where most impactful while preserving computational efficiency elsewhere.

Indexing and Retrieval: The Engram Lookup Pipeline

Token-to-Slot Hashing

Before retrieval, token windows must be mapped to Engram slots. The engram_indices() function in needle/model/architecture.py (lines 146-155) performs deterministic hashing:

from needle.model.architecture import engram_indices

indices = engram_indices(
    tokens,           # Input token IDs

    orders=(2, 3),    # Bigram and trigram features

    heads=8,
    slots=64
)

This function computes hash codes that distribute token sequences across the available slots, enabling O(1) lookup time regardless of sequence length.

KV Extraction in the Forward Pass

During inference or training, the model extracts Engram KV pairs through the _engram_kv method. Here's how to access these tensors programmatically:

def forward(tokens, mask=None):
    # `model.stack` returns hidden states and optional Engram KV.

    hidden, engram_kv = model.stack(
        tokens, 
        mask=mask,
        engram_kv=model._engram_kv(tokens, mask, quant=False)
    )

    # `engram_kv` is a tuple (k, v) for inspection or downstream use.

    k, v = engram_kv
    print("Engram keys shape:", k.shape)   # (batch, sites, d_model)

    print("Engram values shape:", v.shape) # (batch, sites, d_model)

    return hidden

For advanced use cases, you can manually query an Engram table:

# Assuming `e` is an instantiated Engram module:

k, v = e(
    indices,
    ngram_ok=jnp.ones_like(indices[..., :1]),   # Enable all n-grams

    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)),       # Enable convolution taps

    quant=False
)

Site-Specific Gating: Blending Engram Memory with Hidden States

The critical integration point occurs in Block.__call__ at needle/model/architecture.py (lines 318-325). Here, Needle 2 implements learned gating that controls how Engram values influence the current hidden state:

# Pseudocode based on lines 320-325:

alpha = nn.sigmoid(
    jnp.einsum("btd,sbtd->sbt", hidden_state, engram_key) 
    / sqrt(self.d_model)
)
augmented_hidden = hidden_state + alpha * engram_value

This mechanism provides three key properties:

1. Similarity-based weighting — The gate alpha scales with dot-product similarity between the input state and Engram key
2. Site-specific control — Each Engram-enabled layer learns independent gating behavior
3. Preserved attention pathway — The main transformer flow (RMS-norm → multi-head attention → Hadamard-MLP) remains intact

The full block flow continues at lines 326-338, demonstrating that Engram acts as an additive memory boost rather than a replacement for standard attention.

Integration with Standard Transformer Attention

After the Engram-augmented addition, the Block proceeds through its remaining operations:

• RMS normalization — Stabilizes hidden states
• Multi-head attention — Standard self-attention over current sequence
• Hadamard MLP — Feedforward transformation

This sequencing ensures that Engram KV memory functions as extra context that enriches representations before they undergo standard attention processing. The architecture thus achieves a hybrid: efficient O(1) retrieval from Engram tables plus expressive O(n²) self-attention where needed.

Where Engram Memory Is Defined and Used

File	Key Components	Lines
needle/model/architecture.py	Engram class, engram_geometry(), engram_indices(), Block.__call__ integration	81-106, 120-124, 146-155, 318-338
needle/model/decode.py	Runtime _engram_kv extraction from model parameters	Throughout
needle/model/export.py	Serialization of Engram tables and projections	Throughout

Performance and Design Trade-offs

The Engram Key-Value Memory architecture offers several advantages over alternatives:

• Computational efficiency — Hash-based lookup avoids quadratic attention costs for long-range dependencies
• Modular placement — Sites can be enabled selectively based on task requirements
• Preserved compatibility — Standard attention mechanisms remain unchanged, simplifying analysis and debugging

However, the approach requires careful tuning of engram_slots to balance collision resistance against memory overhead, and the convolutional taps add minor computational cost for temporal coherence.

Summary

• Engram KV Memory is a neural table-lookup system that caches key-value pairs at selected transformer layers in Needle 2
• Configuration through ModelConfig parameters (engram_layers, engram_slots, engram_orders) controls memory capacity and placement
• The Engram class in needle/model/architecture.py (lines 81-106) implements embedding tables, projections, and convolutional taps
• engram_indices() provides deterministic hashing from token windows to slot indices (lines 146-155)
• Site-specific gating via learned alpha coefficients blends Engram values with hidden states (lines 320-325)
• Integration preserves standard transformer flow while adding O(1) retrieval access to condensed contextual representations

Frequently Asked Questions

What makes Engram different from standard KV caching in transformers?

Standard KV caching stores key-value projections computed during self-attention for reuse in autoregressive generation. Engram KV Memory is a learned, permanent store indexed by token content rather than position—it retrieves contextual information based on what tokens appear, not where they appear. This enables accessing semantically similar contexts from training or earlier in the sequence without recomputing attention.

How does Needle 2 decide which layers should have Engram sites?

The engram_layers configuration parameter explicitly lists layers (e.g., (2, 15)). According to the Needle 2 source code, placement is task-dependent: earlier layers may capture lexical patterns, while deeper layers handle abstract semantic associations. The architecture permits any subset of layers, including all or none.

Can Engram memory be used with quantized inference?

Yes. The _engram_kv method accepts a quant parameter that controls whether retrieved values undergo quantization. When quant=True, Engram tables use compressed representations, reducing memory bandwidth at potential precision cost. The quantization pathway is implemented in needle/model/decode.py.

What are "convolutional taps" in the Engram module?

Convolutional taps are short-range temporal filters applied to Engram outputs before projection to keys and values. They provide local context around each retrieved slot, smoothing hash collisions and capturing sub-token positional information. The tap kernel size is controlled by ENGRAM_CONV_TAPS constant in the codebase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}