# Understanding Engram Key-Value Memory in Needle 2's Transformer Architecture

> Discover Engram Key-Value Memory in Needle 2. This neural cache enhances transformer attention by storing and retrieving context, enabling efficient long-range signal access without altering core mechanics.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-01

---

**The Engram Key-Value (KV) Memory in Needle 2 is a dedicated neural cache that augments standard transformer attention by storing and retrieving contextual information at specific layers, enabling efficient access to long-range or auxiliary signals without modifying core attention mechanics.**

Needle 2, an open-source transformer architecture developed by Cactus Compute, introduces a novel memory mechanism called **Engram** that operates as a plug-in key-value store at selected transformer layers. This article examines how the Engram KV memory works, its integration points in the codebase, and why it represents a significant architectural enhancement over traditional attention-only models.

## What Is Engram Key-Value Memory?

At its core, **Engram Key-Value Memory** is a small neural table-lookup system that maintains learned embedding tables for key-value retrieval. Unlike standard transformer attention, which recomputes key-value projections from hidden states at every layer, Engram pre-computes and caches KV pairs that can be rapidly accessed during forward passes.

The system is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) where the `Engram` class (lines 81-106) implements the core lookup mechanism:

- **Embedding tables** — learned parameters indexed by token hash
- **Dense projections** — transform embeddings into key (`k`) and value (`v`) tensors
- **Convolutional taps** — optional short-range temporal context mixed into each site

## How Engram Geometry Is Configured

The shape of the Engram memory is determined by `engram_geometry()` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 120-124). This helper derives four critical dimensions from the model configuration:

| Dimension | Configuration Parameter | Purpose |
|-----------|------------------------|---------|
| Tables | `engram_orders` | Number of n-gram hash tables |
| Heads | `engram_heads` | Parallel attention heads for Engram lookup |
| Slots | `engram_slots` | Hash bucket capacity per table |
| Sub-dimension | auto-derived | Per-head feature size |

These parameters control the memory capacity and retrieval granularity. Higher `engram_orders` enable capturing more complex token patterns, while `engram_slots` determines collision resistance in the hash-based indexing scheme.

### Enabling Engram Sites in Model Configuration

```python
import needle
from needle.model.architecture import ModelConfig

# Enable Engram memory at layers 2 and 15, with 64 slots each.

cfg = ModelConfig(
    d_model=1024,
    num_layers=24,
    engram_layers=(2, 15),
    engram_slots=64,
    engram_orders=(2, 3),        # n-gram orders

    engram_heads=0,              # auto-derived from d_model

)

model = needle.model.Model(cfg)   # Internally creates Engram tables

```

The `engram_layers` tuple specifies which transformer layers receive the Engram KV injection. This selective placement allows the model to benefit from external memory where most impactful while preserving computational efficiency elsewhere.

## Indexing and Retrieval: The Engram Lookup Pipeline

### Token-to-Slot Hashing

Before retrieval, token windows must be mapped to Engram slots. The `engram_indices()` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 146-155) performs deterministic hashing:

```python
from needle.model.architecture import engram_indices

indices = engram_indices(
    tokens,           # Input token IDs

    orders=(2, 3),    # Bigram and trigram features

    heads=8,
    slots=64
)

```

This function computes hash codes that distribute token sequences across the available slots, enabling O(1) lookup time regardless of sequence length.

### KV Extraction in the Forward Pass

During inference or training, the model extracts Engram KV pairs through the `_engram_kv` method. Here's how to access these tensors programmatically:

```python
def forward(tokens, mask=None):
    # `model.stack` returns hidden states and optional Engram KV.

    hidden, engram_kv = model.stack(
        tokens, 
        mask=mask,
        engram_kv=model._engram_kv(tokens, mask, quant=False)
    )

    # `engram_kv` is a tuple (k, v) for inspection or downstream use.

    k, v = engram_kv
    print("Engram keys shape:", k.shape)   # (batch, sites, d_model)

    print("Engram values shape:", v.shape) # (batch, sites, d_model)

    return hidden

```

For advanced use cases, you can manually query an Engram table:

```python

# Assuming `e` is an instantiated Engram module:

k, v = e(
    indices,
    ngram_ok=jnp.ones_like(indices[..., :1]),   # Enable all n-grams

    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)),       # Enable convolution taps

    quant=False
)

```

## Site-Specific Gating: Blending Engram Memory with Hidden States

The critical integration point occurs in `Block.__call__` at [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 318-325). Here, Needle 2 implements **learned gating** that controls how Engram values influence the current hidden state:

```python

# Pseudocode based on lines 320-325:

alpha = nn.sigmoid(
    jnp.einsum("btd,sbtd->sbt", hidden_state, engram_key) 
    / sqrt(self.d_model)
)
augmented_hidden = hidden_state + alpha * engram_value

```

This mechanism provides three key properties:

1. **Similarity-based weighting** — The gate `alpha` scales with dot-product similarity between the input state and Engram key
2. **Site-specific control** — Each Engram-enabled layer learns independent gating behavior
3. **Preserved attention pathway** — The main transformer flow (RMS-norm → multi-head attention → Hadamard-MLP) remains intact

The full block flow continues at lines 326-338, demonstrating that Engram acts as an **additive memory boost** rather than a replacement for standard attention.

## Integration with Standard Transformer Attention

After the Engram-augmented addition, the `Block` proceeds through its remaining operations:

- **RMS normalization** — Stabilizes hidden states
- **Multi-head attention** — Standard self-attention over current sequence
- **Hadamard MLP** — Feedforward transformation

This sequencing ensures that Engram KV memory functions as **extra context** that enriches representations before they undergo standard attention processing. The architecture thus achieves a hybrid: efficient O(1) retrieval from Engram tables plus expressive O(n²) self-attention where needed.

## Where Engram Memory Is Defined and Used

| File | Key Components | Lines |
|------|---------------|-------|
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | `Engram` class, `engram_geometry()`, `engram_indices()`, `Block.__call__` integration | 81-106, 120-124, 146-155, 318-338 |
| [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) | Runtime `_engram_kv` extraction from model parameters | Throughout |
| [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) | Serialization of Engram tables and projections | Throughout |

## Performance and Design Trade-offs

The **Engram Key-Value Memory** architecture offers several advantages over alternatives:

- **Computational efficiency** — Hash-based lookup avoids quadratic attention costs for long-range dependencies
- **Modular placement** — Sites can be enabled selectively based on task requirements
- **Preserved compatibility** — Standard attention mechanisms remain unchanged, simplifying analysis and debugging

However, the approach requires careful tuning of `engram_slots` to balance collision resistance against memory overhead, and the convolutional taps add minor computational cost for temporal coherence.

## Summary

- **Engram KV Memory** is a neural table-lookup system that caches key-value pairs at selected transformer layers in Needle 2
- Configuration through `ModelConfig` parameters (`engram_layers`, `engram_slots`, `engram_orders`) controls memory capacity and placement
- The `Engram` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 81-106) implements embedding tables, projections, and convolutional taps
- `engram_indices()` provides deterministic hashing from token windows to slot indices (lines 146-155)
- Site-specific gating via learned `alpha` coefficients blends Engram values with hidden states (lines 320-325)
- Integration preserves standard transformer flow while adding O(1) retrieval access to condensed contextual representations

## Frequently Asked Questions

### What makes Engram different from standard KV caching in transformers?

Standard KV caching stores key-value projections computed during self-attention for reuse in autoregressive generation. **Engram KV Memory** is a learned, permanent store indexed by token content rather than position—it retrieves contextual information based on what tokens appear, not where they appear. This enables accessing semantically similar contexts from training or earlier in the sequence without recomputing attention.

### How does Needle 2 decide which layers should have Engram sites?

The `engram_layers` configuration parameter explicitly lists layers (e.g., `(2, 15)`). According to the Needle 2 source code, placement is task-dependent: earlier layers may capture lexical patterns, while deeper layers handle abstract semantic associations. The architecture permits any subset of layers, including all or none.

### Can Engram memory be used with quantized inference?

Yes. The `_engram_kv` method accepts a `quant` parameter that controls whether retrieved values undergo quantization. When `quant=True`, Engram tables use compressed representations, reducing memory bandwidth at potential precision cost. The quantization pathway is implemented in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py).

### What are "convolutional taps" in the Engram module?

**Convolutional taps** are short-range temporal filters applied to Engram outputs before projection to keys and values. They provide local context around each retrieved slot, smoothing hash collisions and capturing sub-token positional information. The tap kernel size is controlled by `ENGRAM_CONV_TAPS` constant in the codebase.