Engram Key-Value Memory: Implementing Contextual Recall in Needle

The Engram module in the Needle architecture implements a key-value memory system that stores compressed n-gram embeddings, enabling efficient long-range contextual recall through gated integration with standard attention mechanisms.

The Engram Key-Value Memory system is a specialized component within the cactus-compute/needle repository designed to enhance transformer architectures with persistent contextual memory. By maintaining compressed representations of recent token n-grams, this mechanism allows models to recall distant dependencies without re-encoding entire sequences. The implementation centers on a learnable embedding table that projects stored contexts into key-value pairs fused during multi-head attention.

How Engram Key-Value Memory Works

The Engram class defined in needle/model/architecture.py (lines 181-190) creates a persistent memory structure that operates alongside standard transformer layers. During the forward pass, the system maps token indices to slots in embedding tables and transforms these representations into keys and values compatible with the attention mechanism.

N-Gram Embedding Storage

At the core of the Engram Key-Value Memory is a set of embedding tables configured via hyperparameters including num_tables, slots, and sub_dim. According to lines 192-197 of architecture.py, the module retrieves vectors from these tables using prepared indices, reshaping them into a combined key-value tensor. This design enables the model to maintain a compressed history of seen n-grams across multiple granularities.

Key-Value Projection and Temporal Smoothing

Following embedding retrieval, the combined tensor undergoes projection into separate key (k) and value (v) vectors through learned projections k_proj and v_proj (lines 199-202). The implementation additionally applies convolutional taps to the value stream using configurable dilation parameters (lines 203-206), enhancing temporal smoothing and stabilizing the memory representations across sequence positions.

Integrating Engram Memory into Transformer Blocks

The integration point occurs within the Block.__call__ method (lines 319-328), where the Transformer block receives an optional engram_kv argument containing the Engram-derived keys and values.

When provided, the Block performs a gated addition that blends the Engram values (ev) into the current hidden state. A learned gating factor controls the interpolation between the standard attention output and the retrieved contextual memory. This mechanism enables the model to dynamically weight historical context against current token processing, improving performance on tasks requiring long-range dependency tracking.

Implementation Details and Source Code

The Engram implementation spans multiple components within the Needle codebase:

• needle/model/architecture.py (lines 181-206): Contains the Engram class definition, embedding lookups, and projection logic
• needle/model/architecture.py (lines 319-328): Houses the Block.__call__ integration where Engram KV pairs merge with hidden states
• needle/model/run.py: Orchestrates high-level model execution, building Engram components and feeding KV pairs to the network
• needle/model/__init__.py: Exports the Engram module for external consumption

The embedding parameters are initialized via self.param("embedding", ...) calls, creating trainable tables that the optimizer updates during backpropagation alongside standard attention weights.

Practical Usage Examples

Instantiating an Engram Key-Value Memory layer requires specifying model dimensions and table configurations:

engram = Engram(
    d_model=768,          # model dimension

    num_tables=12,        # number of embedding tables

    slots=1024,           # size of each table

    sub_dim=64,           # embedding sub-dimension

    num_layers=12,
    conv_dilation=2,
)
indices, ngram_ok, tap_ok = ...  # prepared by the tokeniser

k, v = engram(indices, ngram_ok, tap_ok, quant=False)

When processing through a Transformer block, pass the Engram KV tuples via the engram_kv parameter:

# Assume `x` is the current hidden representation

# `engram_kv` is the tuple (engram_keys, engram_values) from the Engram module

output = Block(
    num_heads=8,
    num_kv_heads=8,
    d_model=768,
    num_layers=12,
)(x, mask=mask, rope=rope, quant=False, engram_kv=engram_kv, site_flags=flags)

For end-to-end inference with contextual recall:

model = NeedleModel(...)                      # high-level model wrapper

engram = model.engram                         # internal Engram instance

indices, ngram_ok, tap_ok = model.prepare(...)

# Retrieve Engram KV pairs

engram_k, engram_v = engram(indices, ngram_ok, tap_ok)

# Forward pass through the network, providing Engram KV for contextual recall

logits = model(x, engram_kv=(engram_k, engram_v))

Summary

• Engram Key-Value Memory stores compressed n-gram embeddings in learnable tables defined in needle/model/architecture.py, enabling efficient contextual recall without full sequence re-encoding.
• The system projects stored representations into keys and values (lines 199-202) and applies convolutional temporal smoothing (lines 203-206) before injection into attention layers.
• Transformer blocks integrate Engram memory through gated addition in Block.__call__ (lines 319-328), allowing dynamic weighting of historical context against current inputs.
• Configuration parameters including num_tables, slots, and sub_dim control the memory capacity and compression characteristics of the Engram layer.

Frequently Asked Questions

What is the primary purpose of Engram Key-Value Memory in the Needle architecture?

The Engram Key-Value Memory serves as a persistent storage mechanism for compressed n-gram representations, allowing the model to recall distant contextual information efficiently. Unlike standard attention which recomputes relationships from scratch, the Engram retrieves pre-computed embeddings from tables in needle/model/architecture.py, reducing computational overhead for long-range dependencies.

How does the Engram module integrate with standard Transformer blocks?

Integration occurs within the Block.__call__ method at lines 319-328 of architecture.py, where the block accepts an optional engram_kv tuple containing keys and values. The implementation performs a gated addition that blends Engram-derived values (ev) into the hidden state using a learned gating factor, effectively interpolating between current attention outputs and retrieved historical context.

What configuration parameters control Engram memory capacity?

The Engram class accepts several hyperparameters that determine memory characteristics: num_tables specifies the number of parallel embedding tables, slots defines the size of each table, and sub_dim controls the embedding sub-dimension. Additionally, conv_dilation configures the temporal smoothing applied to value projections, affecting how the model aggregates information across sequence positions.

Can Engram Key-Value Memory be used with quantization?

Yes, the Engram forward pass supports quantization via the quant parameter. When invoking engram(indices, ngram_ok, tap_ok, quant=False), setting quant=True enables compressed numerical representations of the embedding tables and projected key-value pairs, reducing memory footprint during inference while maintaining contextual recall capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}