# Engram Key-Value Memory in Needle 2: Architecture and Implementation

> Discover Engram key-value memory in Needle 2 a hashed n-gram storage system that boosts transformer blocks with persistent representations for constant RAM tool calling.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-22

---

**Engram key-value memory in Needle 2 is a fixed-size hashed n-gram storage system that augments transformer blocks with persistent, bounded-memory representations of recent tokens, enabling constant-RAM tool calling across long conversations.**

Needle 2 augments its attention mechanism with an **engram key-value (KV) memory** that stores hashed n-gram representations of the recent token stream. Unlike standard KV caches that grow linearly with sequence length, this mechanism maintains a constant footprint of approximately 28 MiB regardless of interaction duration. The implementation centers on the `Engram` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and integrates directly into the forward pass of specific transformer layers.

## What Is Engram Key-Value Memory?

The engram KV memory consists of a set of fixed-size tables (`embedding`) that hold vector slots. For every incoming token, a set of indices is derived from the token’s n-gram context (`engram_indices`). Those indices address the tables, producing a *key* (`k`) and a *value* (`v`) vector for each site. The values are further refined by a small convolution (“tap”) that mixes neighboring positions.

This design lets the model retain a compact, sliding-window memory of past tokens while keeping the total footprint bounded. As documented in the [README](https://github.com/cactus-compute/needle/blob/main/README.md#L13), the memory cost remains constant regardless of conversation length because the tables are fixed-size and the sliding window is limited to 256 tokens. The core data structures are defined in lines 81-106 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

## Where Engram KV Memory Fires in the Architecture

The engram KV memory activates at specific injection points throughout the model stack rather than at every layer.

### Layer Configuration via `engram_layers`

Developers specify which transformer layers receive engram memory through the `engram_layers` configuration tuple. Typical configurations found in [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py) inject the memory at strategic depths—such as layers `(2, 8, 14)`—to balance early context acquisition with deep semantic processing.

### Block-Level Integration in `Block.__call__`

Inside each configured `Block`, the memory fires within the `__call__` method (lines 18-27 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)). The block receives an optional `engram_kv` tuple containing the pre-computed keys and values. If present, the block computes an attention-style gating (`alpha`) between the current hidden state and the retrieved engram values, then adds the gated contribution to the block’s hidden representation (`x = x + ...`). This occurs before the regular self-attention and Hadamard-MLP steps.

### Decode Pipeline Wiring

The [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) file contains the helper `_engram_kv` that builds the KV tuple for each forward pass, orchestrating the flow from index computation to block injection. This separation allows the model to pre-compute engram indices while maintaining clean abstractions between memory retrieval and consumption.

## Technical Implementation Details

### Index Computation with `_engram_indices`

The `_engram_indices` function transforms the token list into table indices based on configurable hyperparameters: `engram_orders` (the n-gram sizes), `engram_heads` (number of attention heads per order), and `engram_slots` (hash table size). This hashing mechanism compresses variable-length contexts into fixed-size addresses.

### Table Lookup and Projection

Inside `Engram.__call__`, the implementation performs the following operations:

1. **Table indexing**: The tables are indexed via `tables[jnp.arange(self.num_tables), indices]`
2. **Linear projection**: Retrieved vectors pass through lightweight dense layers (`key_proj`, `value_proj`) to produce the final key and value embeddings
3. **Tap convolution**: Values undergo blending through the tap convolution (`taps` + `_shift_right`), which mixes information from neighboring positions to smooth local context

## Working with Engram KV Memory: Code Examples

### Basic Usage with Default Configuration

```python
import needle

# Create a Needle agent with the built‑in engram KV memory (enabled by default)

agent = needle.Needle(tools=[...])          # tools are described with @needle.tool

response = agent.run("Summarize the last three tool calls.") 
print(response["results"])

```

### Customizing Memory Capacity and Injection Sites

```python
from needle import Needle, config

# Adjust the engram hyper‑parameters

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # sites where engram KV is injected

    engram_slots=16384,         # size of each hash table

    engram_orders=(2, 3, 4),   # n‑gram orders used for indexing

    engram_heads=4,             # number of heads per order

)

agent = Needle(tools=[...], config=custom_cfg)
print(agent.run("What was the last tool you called?")["results"])

```

### Low-Level Manual Retrieval

```python
import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Assume we have a token array `tokens` and a compiled model `mdl`

cfg = agent.config
orders, heads, _ = engram_geometry(cfg)
indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

# Retrieve the KV pair from the first engram site

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(orders) * heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

k, v = engram(indices, ngram_ok=jnp.ones(indices.shape[:-1]), tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), quant=False)

# `k` and `v` can now be passed to a Block as `engram_kv=(k, v)`

```

## Memory Bounds and Performance Characteristics

The engram KV memory provides a **persistent KV sink** for tool-related tokens, enabling the model to remember tool calls across a long interaction without growing RAM usage. Because the tables are fixed-size and the sliding window is limited to 256 tokens, the memory cost remains constant at approximately 28 MiB regardless of conversation length. This bounded approach contrasts sharply with standard transformer KV caches that consume GPU memory proportional to sequence length multiplied by batch size and head dimensions.

## Summary

- **Engram KV memory** uses fixed-size hash tables to store n-gram representations with a constant footprint of ~28 MiB
- The system **fires at configured layers** specified by `engram_layers` and integrates inside `Block.__call__` (lines 18-27 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)) via attention-style gating
- **Index computation** occurs via `_engram_indices` using `engram_orders`, `engram_heads`, and `engram_slots` parameters
- Retrieved values undergo **tap convolution** for position mixing before being projected through `key_proj` and `value_proj` layers
- This architecture enables **long-horizon tool calling** without the linear memory growth associated with traditional KV caches

## Frequently Asked Questions

### What is the fixed memory footprint of Engram KV memory in Needle 2?

The engram KV memory maintains a constant footprint of approximately 28 MiB regardless of conversation length. This bounded size results from fixed-size hash tables and a 256-token sliding window, as implemented in the `Engram` class (lines 81-106 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)).

### How does Engram KV memory differ from a standard transformer KV cache?

Standard transformer KV caches grow linearly with sequence length, consuming additional memory for every new token. Engram KV memory uses hashed n-gram indexing into fixed-size tables, providing persistent context storage with constant memory usage independent of sequence length.

### Where exactly does Engram KV memory integrate with the transformer blocks?

The memory integrates inside `Block.__call__` at lines 18-27 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), occurring before standard self-attention. It receives the `engram_kv` tuple computed by `_engram_kv` in [`decode.py`](https://github.com/cactus-compute/needle/blob/main/decode.py), applies attention-style gating (`alpha`), and adds the gated contribution to the hidden state.

### Can developers customize the n-gram hashing behavior for Engram memory?

Yes. Developers control the hashing via the `engram_orders`, `engram_heads`, and `engram_slots` configuration parameters. These settings determine which n-gram sizes are hashed, how many parallel heads process each order, and the size of the underlying hash tables, allowing trade-offs between collision resistance and memory consumption.