# How the Engram Key-Value Memory Works in Needle 2 Architecture

> Discover how Needle 2's Engram Key-Value Memory uses learned hash tables for deterministic n-gram hashing, enabling arbitrarily long sequences with bounded memory.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-09-09

---

**The Engram Key-Value Memory in Needle 2 replaces the linearly-growing Transformer KV cache with a fixed-size, learned hash table that retrieves long-range context via deterministic n-gram hashing, enabling arbitrarily long sequences with bounded memory.**

Needle 2, maintained in the `cactus-compute/needle` repository, implements a sparse, high-capacity memory system called the Engram that stores attention keys and values outside the traditional token-by-token cache. Unlike standard Transformers where memory scales with sequence length, the **Engram Key-Value Memory** uses learned embeddings indexed by hashed n-grams to provide constant-time access to semantic context from any position in the input.

## Geometry and Configuration of the Engram

The memory layout is defined by three hyper-parameters stored in the model configuration:

- `engram_orders` – tuple defining n-gram orders (e.g., `(2, 3)`)
- `engram_heads` – number of heads per order (defaults to `⌈d_model / (len(orders) × sub_dim)⌉`)
- `engram_slots` – total hash table size (defaults to `8192`)

### Computing Memory Dimensions

The helper function `engram_geometry` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) calculates the final geometry used to initialize embedding tables:

```python

# Lines 20-25 in needle/model/architecture.py

def engram_geometry(orders, d_model, sub_dim):
    """Compute (orders, heads_per_order, sub_dim) for Engram tables."""
    num_orders = len(orders)
    heads = math.ceil(d_model / (num_orders * sub_dim))
    return orders, heads, sub_dim

```

This geometry determines the shape of the learned memory banks that will store the key-value pairs.

## Token-to-Slot Mapping via Deterministic Hashing

For every input sequence, Needle 2 computes hash indices that map token windows to specific slots in the Engram tables. This process is handled by the `engram_indices` function.

### The engram_indices Implementation

Located at lines 46-58 in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this function builds a deterministic hash for each (order, head) pair:

```python
indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

```

The implementation uses a multiply-add hash combining `_ENGRAM_SEED` and `_ENGRAM_PRIME` constants. It mixes token IDs with shifted versions of the token stream (up to the order length) and reduces the result modulo `slots`, producing an integer index per token per head. This deterministic approach ensures that identical n-grams always map to the same memory location across different forward passes.

## Embedding Table Look-up and KV Projection

Each `Engram` module owns a learned embedding table shaped `(num_tables, slots, sub_dim)`, initialized in the module's constructor at lines 81-88 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

### Memory Retrieval and Masking

During the forward pass, the model retrieves vectors using the pre-computed indices:

```python
tables = self.param("embedding", default_init(),
                    (self.num_tables, self.slots, self.sub_dim))
fetched = tables[jnp.arange(self.num_tables), indices]

# Shape: (B, T, num_tables, sub_dim)

```

The retrieved vectors pass through two binary masks before becoming valid keys and values:

- **`ngram_ok`** – Enforced in `_engram_kv` (lines 14-16), this mask zeros out entries that would cross token-order boundaries, ensuring the n-gram window is respected.
- **`tap_ok`** – Implemented in `Engram.__call__` (lines 102-106), this enables a small convolutional "tap" filter defined by `ENGRAM_CONV_TAPS` that mixes recent values spatially.

### Projection and Quantization

After masking and reshaping (`e = fetched.reshape(*indices.shape[:2], self.num_tables * self.sub_dim)`), the vectors are linearly projected to the model dimension:

```python
k = nn.Dense(self.d_model)(e)   # Key projection

v = nn.Dense(self.d_model)(e)   # Value projection

```

Both projections support optional quantization via the `_aq` parameter for mixed-precision inference, as seen in `Engram.__call__` at lines 95-98.

## Integration into the Transformer Stack

The `SimpleAttentionNetwork` builds Engram modules for layers specified in the configuration (`engram_layers`), instantiating them in `__init__` at lines 90-95.

### KV Retrieval Pipeline

The internal `_engram_kv` method (lines 13-19) executes the retrieval workflow:

1. Computes hash indices via `engram_indices`.
2. Builds `ngram_ok` and `tap_ok` masks.
3. Invokes each Engram module to obtain `(k, v)` tuples.
4. Stacks results into tensors shaped `(num_engram_layers, B, T, d_model)`.

### Gated Merge in Transformer Blocks

Inside each `Block`, the Engram KV pairs merge with token representations via a learned gating mechanism implemented in `Block.__call__` at lines 19-25:

```python
alpha = nn.sigmoid(
    jnp.einsum("btd,sbtd->sbt", _rms_unit(x), _rms_unit(ek))
    / math.sqrt(self.d_model)
)
x = x + jnp.einsum("s,sbt,sbtd->btd", site_flags, alpha, ev)

```

This sigmoid-controlled gate allows each layer to dynamically weight the outside-the-window context retrieved from the Engram memory against the local token representation.

## Memory Efficiency and Long-Range Context

The Engram Key-Value Memory offers distinct architectural advantages over traditional caching:

**Memory size** – Traditional caches grow linearly with sequence length (tokens × heads), while the Engram maintains a fixed footprint of `slots × sub_dim` regardless of context length.

**Long-range recall** – Standard KV caches are limited to recent context unless the full history is retained. The Engram retrieves from any past position via hash collision, enabling arbitrarily long context windows.

**Computational cost** – Instead of O(N²) attention over growing caches, the Engram requires O(1) constant-time lookups plus cheap linear projections.

**Sparsity** – Unlike dense traditional caches that store all tokens, the Engram learns sparse access patterns where only hashed slots contain information.

**Learnability** – While standard KV caches derive directly from token embeddings, Engram tables are fully learnable with convolutional taps, allowing the model to store compressed semantic memory rather than literal token representations.

## End-to-End Implementation Example

The following example demonstrates configuring and running inference with the Engram Key-Value Memory enabled:

```python
import jax.numpy as jnp
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# Configure model with Engram layers active at positions 2-15

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    engram_layers=(2, 15),          # Tuple specifying active layer range

    engram_orders=(2, 3),
    engram_slots=8192,
)

model = SimpleAttentionNetwork(cfg)

# Dummy input: batch_size=1, seq_len=10

tokens = jnp.arange(10)[None, :]

# Forward pass automatically uses Engram KV memory

logits = model(tokens)

# Inspect the generated Engram KV tensors

engram_kv = model._engram_kv(tokens, mask=None, quant=False)
keys, values = engram_kv
print(keys.shape, values.shape)   # (num_engram_layers, 1, 10, d_model)

```

In this implementation, [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) transparently handles the Engram initialization, hashing, and integration, while `SimpleAttentionNetwork` orchestrates the retrieval pipeline defined in `_engram_kv`.

## Summary

- **Fixed-size storage** – The Engram uses hash tables with configurable `slots` (default 8192) independent of sequence length, solving the linear memory growth problem of standard Transformers.
- **Deterministic hashing** – The `engram_indices` function maps token n-grams to slots using seeded multiply-add hashing in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- **Learned embeddings** – Each Engram module maintains learnable tables shaped `(num_tables, slots, sub_dim)` with optional convolutional taps for local context mixing.
- **Gated integration** – Transformer blocks merge Engram KV pairs via sigmoid gating in `Block.__call__`, allowing dynamic weighting of long-range context.
- **Quantization support** – The architecture supports mixed-precision training and inference through optional `_aq` quantization on projection layers.

## Frequently Asked Questions

### How does the Engram Key-Value Memory differ from standard Transformer KV caching?

Traditional KV caches store every token's key and value vectors in a linear buffer that grows with sequence length, consuming O(N) memory. The Engram Key-Value Memory compresses context into fixed-size hash tables accessed by n-gram hashes, maintaining O(1) memory usage regardless of input length while enabling retrieval from arbitrarily distant positions.

### What prevents hash collisions from corrupting the Engram memory?

The `engram_indices` function uses deterministic hashing modulo `engram_slots`, meaning collisions are intentional and frequency-based. The learned embedding tables adapt during training to store superposed representations in collision slots, effectively learning to compress multiple n-gram contexts into shared memory locations through gradient descent, similar to how standard embedding layers handle vocabulary hashing.

### Which transformer layers utilize the Engram memory?

The `engram_layers` parameter in `TransformerConfig` specifies a tuple `(start, end)` defining which layer indices receive Engram KV pairs. According to the `SimpleAttentionNetwork` initialization in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), layers outside this range operate with standard self-attention only, allowing architectures to mix local attention (early/late layers) with Engram-augmented attention (middle layers).

### Is the Engram Key-Value Memory compatible with quantization?

Yes, the Engram implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) supports mixed-precision inference. The `Engram.__call__` method accepts a `quant` parameter that triggers `_aq` (activation quantization) on the dense projections that convert retrieved embeddings into keys and values, reducing memory bandwidth during retrieval without altering the hash table storage format.