# Engram Key-Value Memory in Needle 2 Architecture: Technical Deep Dive

> Explore Engram key-value memory in Needle 2 architecture. Discover this fixed-size, hashed n-gram storage that adds persistent memory to attention layers, using only 28 MiB.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-12

---

**Engram key-value memory in Needle 2 is a fixed-size, hashed n-gram storage mechanism that augments the attention layer with a bounded persistent memory for tool-related tokens, consuming approximately 28 MiB regardless of conversation length.**

The Needle 2 open-source agent framework introduces a novel memory architecture to handle long-running tool interactions without unbounded RAM growth. At the core of this design sits the **engram key-value memory**, a compact, sliding-window storage system that hashes recent token n-grams into fixed-size lookup tables. Unlike standard KV caches that grow linearly with sequence length, this mechanism maintains constant memory usage while providing persistent context for function calls across extended interactions.

## What Is Engram Key-Value Memory?

The engram key-value memory consists of fixed-size embedding tables that store vector representations of recent token contexts. For every incoming token, the system derives indices from the token's n-gram context using the `engram_indices` function, addressing slots within hash tables to produce key (`k`) and value (`v`) vectors. These values undergo further refinement through a lightweight convolution operation ("tap") that mixes neighboring positions, creating a localized, compressed memory of the recent token stream.

According to the [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) and the implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this design caps memory consumption at approximately 28 MiB. The bounded nature stems from fixed table sizes and a sliding window limited to 256 tokens, ensuring that RAM usage remains constant even during extended multi-turn conversations with numerous tool invocations.

## How Engram KV Memory Works

The implementation spans three distinct phases: index computation, table retrieval with value refinement, and block-level integration.

### Index Computation

The `_engram_indices` function transforms incoming token sequences into table indices based on configurable hyperparameters. These include `engram_orders` (defining n-gram sizes), `engram_heads` (parallel hashing functions per order), and `engram_slots` (table size). By hashing n-gram contexts, the system maps variable-length token histories to fixed-size table addresses, enabling constant-time lookups regardless of conversation depth.

### Table Lookup and Value Refinement

Within the `Engram` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `__call__` method performs the actual memory retrieval. It indexes the embedding tables using `tables[jnp.arange(self.num_tables), indices]`, then projects the retrieved vectors into keys and values via dedicated `key_proj` and `value_proj` layers. The values receive additional processing through the tap convolution—implemented using `taps` and `_shift_right` operations—which blends spatially adjacent memory slots to capture local context patterns.

### Integration with Transformer Blocks

Each transformer `Block` optionally accepts an `engram_kv` tuple containing the retrieved key-value pairs. Inside `Block.__call__` (lines 18-27 in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)), the block computes an attention-style gating parameter (`alpha`) that modulates the interaction between the current hidden state and the engram values. The gated contribution adds directly to the block's hidden representation (`x = x + ...`) before the standard self-attention and Hadamard-MLP processing steps.

## Configuration and Usage

Needle 2 enables engram KV memory by default, but developers can customize or access it programmatically through the configuration API and low-level module interfaces.

Basic usage with default settings:

```python
import needle

# Create agent with built-in engram KV memory (enabled by default)

agent = needle.Needle(tools=[...])
response = agent.run("Summarize the last three tool calls.")
print(response["results"])

```

Customizing memory parameters:

```python
from needle import Needle, config

# Adjust engram hyperparameters for specific workloads

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # Injection sites

    engram_slots=16384,         # Hash table size

    engram_orders=(2, 3, 4),    # N-gram orders

    engram_heads=4,             # Heads per order

)

agent = Needle(tools=[...], config=custom_cfg)

```

Low-level manual retrieval:

```python
import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Assuming `tokens` array and compiled model

cfg = agent.config
indices = engram_indices(tokens, cfg.engram_orders, cfg.engram_heads, cfg.engram_slots)

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(cfg.engram_orders) * cfg.engram_heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

k, v = engram(
    indices, 
    ngram_ok=jnp.ones(indices.shape[:-1]), 
    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), 
    quant=False
)

# Pass engram_kv=(k, v) to Block

```

## Summary

- **Engram key-value memory** provides a fixed-size, hashed n-gram storage mechanism in Needle 2, bounding memory usage at approximately 28 MiB regardless of sequence length.
- The system uses **`_engram_indices`** to hash token contexts into table addresses based on configurable orders, heads, and slots defined in [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py).
- Table lookups occur in **`Engram.__call__`** with value refinement via tap convolution before injection into transformer blocks.
- Each **`Block`** integrates engram KV through attention-style gating (`alpha`) that augments hidden states before standard self-attention processing.
- The wiring into the model pipeline is handled by **[`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py)**, which constructs the KV tuple for each forward pass.

## Frequently Asked Questions

### What is the memory footprint of engram KV memory?

The engram key-value memory consumes approximately 28 MiB of RAM regardless of conversation length. This bounded footprint results from fixed-size hash tables and a sliding window limited to 256 tokens, ensuring constant memory usage during long-running agent interactions.

### How does engram memory differ from standard KV caching?

Standard KV caches grow linearly with sequence length, storing every token's key and value vectors. In contrast, engram memory uses hashed n-gram indices to compress historical context into fixed-size tables, providing a persistent memory sink without unbounded growth, specifically optimized for retaining tool call context.

### Can engram memory be disabled or customized?

Yes. While enabled by default, you can customize engram parameters through `config.Config` by adjusting `engram_layers`, `engram_slots`, `engram_orders`, and `engram_heads`. The low-level API in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) also allows manual control over engram computation and injection.

### Which layers support engram KV injection?

Engram KV memory can be injected at arbitrary transformer layers specified by the `engram_layers` tuple (typically layers 2, 8, and 14 in default configurations). Each specified `Block` checks for the `engram_kv` parameter and applies the gated integration when present.