# Engram Key-Value Memory: Implementing Contextual Recall in Needle

> Explore Engram Key-Value Memory in Needle for efficient contextual recall. Learn how n-gram embeddings and gated attention enhance long-range memory.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-04

---

**The Engram module in the Needle architecture implements a key-value memory system that stores compressed n-gram embeddings, enabling efficient long-range contextual recall through gated integration with standard attention mechanisms.**

The **Engram Key-Value Memory** system is a specialized component within the [cactus-compute/needle](https://github.com/cactus-compute/needle) repository designed to enhance transformer architectures with persistent contextual memory. By maintaining compressed representations of recent token n-grams, this mechanism allows models to recall distant dependencies without re-encoding entire sequences. The implementation centers on a learnable embedding table that projects stored contexts into key-value pairs fused during multi-head attention.

## How Engram Key-Value Memory Works

The `Engram` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 181-190) creates a persistent memory structure that operates alongside standard transformer layers. During the forward pass, the system maps token indices to slots in embedding tables and transforms these representations into keys and values compatible with the attention mechanism.

### N-Gram Embedding Storage

At the core of the **Engram Key-Value Memory** is a set of embedding tables configured via hyperparameters including `num_tables`, `slots`, and `sub_dim`. According to lines 192-197 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py), the module retrieves vectors from these tables using prepared indices, reshaping them into a combined key-value tensor. This design enables the model to maintain a compressed history of seen n-grams across multiple granularities.

### Key-Value Projection and Temporal Smoothing

Following embedding retrieval, the combined tensor undergoes projection into separate key (`k`) and value (`v`) vectors through learned projections `k_proj` and `v_proj` (lines 199-202). The implementation additionally applies convolutional taps to the value stream using configurable dilation parameters (lines 203-206), enhancing temporal smoothing and stabilizing the memory representations across sequence positions.

## Integrating Engram Memory into Transformer Blocks

The integration point occurs within the `Block.__call__` method (lines 319-328), where the Transformer block receives an optional `engram_kv` argument containing the Engram-derived keys and values.

When provided, the Block performs a **gated addition** that blends the Engram values (`ev`) into the current hidden state. A learned gating factor controls the interpolation between the standard attention output and the retrieved contextual memory. This mechanism enables the model to dynamically weight historical context against current token processing, improving performance on tasks requiring long-range dependency tracking.

## Implementation Details and Source Code

The Engram implementation spans multiple components within the Needle codebase:

- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** (lines 181-206): Contains the `Engram` class definition, embedding lookups, and projection logic
- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** (lines 319-328): Houses the `Block.__call__` integration where Engram KV pairs merge with hidden states
- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)**: Orchestrates high-level model execution, building Engram components and feeding KV pairs to the network
- **[`needle/model/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/__init__.py)**: Exports the Engram module for external consumption

The embedding parameters are initialized via `self.param("embedding", ...)` calls, creating trainable tables that the optimizer updates during backpropagation alongside standard attention weights.

## Practical Usage Examples

Instantiating an **Engram Key-Value Memory** layer requires specifying model dimensions and table configurations:

```python
engram = Engram(
    d_model=768,          # model dimension

    num_tables=12,        # number of embedding tables

    slots=1024,           # size of each table

    sub_dim=64,           # embedding sub-dimension

    num_layers=12,
    conv_dilation=2,
)
indices, ngram_ok, tap_ok = ...  # prepared by the tokeniser

k, v = engram(indices, ngram_ok, tap_ok, quant=False)

```

When processing through a Transformer block, pass the Engram KV tuples via the `engram_kv` parameter:

```python

# Assume `x` is the current hidden representation

# `engram_kv` is the tuple (engram_keys, engram_values) from the Engram module

output = Block(
    num_heads=8,
    num_kv_heads=8,
    d_model=768,
    num_layers=12,
)(x, mask=mask, rope=rope, quant=False, engram_kv=engram_kv, site_flags=flags)

```

For end-to-end inference with contextual recall:

```python
model = NeedleModel(...)                      # high-level model wrapper

engram = model.engram                         # internal Engram instance

indices, ngram_ok, tap_ok = model.prepare(...)

# Retrieve Engram KV pairs

engram_k, engram_v = engram(indices, ngram_ok, tap_ok)

# Forward pass through the network, providing Engram KV for contextual recall

logits = model(x, engram_kv=(engram_k, engram_v))

```

## Summary

- **Engram Key-Value Memory** stores compressed n-gram embeddings in learnable tables defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), enabling efficient contextual recall without full sequence re-encoding.
- The system projects stored representations into keys and values (lines 199-202) and applies convolutional temporal smoothing (lines 203-206) before injection into attention layers.
- Transformer blocks integrate Engram memory through gated addition in `Block.__call__` (lines 319-328), allowing dynamic weighting of historical context against current inputs.
- Configuration parameters including `num_tables`, `slots`, and `sub_dim` control the memory capacity and compression characteristics of the Engram layer.

## Frequently Asked Questions

### What is the primary purpose of Engram Key-Value Memory in the Needle architecture?

The **Engram Key-Value Memory** serves as a persistent storage mechanism for compressed n-gram representations, allowing the model to recall distant contextual information efficiently. Unlike standard attention which recomputes relationships from scratch, the Engram retrieves pre-computed embeddings from tables in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), reducing computational overhead for long-range dependencies.

### How does the Engram module integrate with standard Transformer blocks?

Integration occurs within the `Block.__call__` method at lines 319-328 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py), where the block accepts an optional `engram_kv` tuple containing keys and values. The implementation performs a gated addition that blends Engram-derived values (`ev`) into the hidden state using a learned gating factor, effectively interpolating between current attention outputs and retrieved historical context.

### What configuration parameters control Engram memory capacity?

The `Engram` class accepts several hyperparameters that determine memory characteristics: `num_tables` specifies the number of parallel embedding tables, `slots` defines the size of each table, and `sub_dim` controls the embedding sub-dimension. Additionally, `conv_dilation` configures the temporal smoothing applied to value projections, affecting how the model aggregates information across sequence positions.

### Can Engram Key-Value Memory be used with quantization?

Yes, the Engram forward pass supports quantization via the `quant` parameter. When invoking `engram(indices, ngram_ok, tap_ok, quant=False)`, setting `quant=True` enables compressed numerical representations of the embedding tables and projected key-value pairs, reducing memory footprint during inference while maintaining contextual recall capabilities.