Engram Key-Value Memory in Needle 2: Architecture and Implementation

Engram key-value memory in Needle 2 is a fixed-size hashed n-gram storage system that augments transformer blocks with persistent, bounded-memory representations of recent tokens, enabling constant-RAM tool calling across long conversations.

Needle 2 augments its attention mechanism with an engram key-value (KV) memory that stores hashed n-gram representations of the recent token stream. Unlike standard KV caches that grow linearly with sequence length, this mechanism maintains a constant footprint of approximately 28 MiB regardless of interaction duration. The implementation centers on the Engram class in needle/model/architecture.py and integrates directly into the forward pass of specific transformer layers.

What Is Engram Key-Value Memory?

The engram KV memory consists of a set of fixed-size tables (embedding) that hold vector slots. For every incoming token, a set of indices is derived from the token’s n-gram context (engram_indices). Those indices address the tables, producing a key (k) and a value (v) vector for each site. The values are further refined by a small convolution (“tap”) that mixes neighboring positions.

This design lets the model retain a compact, sliding-window memory of past tokens while keeping the total footprint bounded. As documented in the README, the memory cost remains constant regardless of conversation length because the tables are fixed-size and the sliding window is limited to 256 tokens. The core data structures are defined in lines 81-106 of needle/model/architecture.py.

Where Engram KV Memory Fires in the Architecture

The engram KV memory activates at specific injection points throughout the model stack rather than at every layer.

Layer Configuration via engram_layers

Developers specify which transformer layers receive engram memory through the engram_layers configuration tuple. Typical configurations found in tests/conftest.py inject the memory at strategic depths—such as layers (2, 8, 14)—to balance early context acquisition with deep semantic processing.

Block-Level Integration in Block.__call__

Inside each configured Block, the memory fires within the __call__ method (lines 18-27 of needle/model/architecture.py). The block receives an optional engram_kv tuple containing the pre-computed keys and values. If present, the block computes an attention-style gating (alpha) between the current hidden state and the retrieved engram values, then adds the gated contribution to the block’s hidden representation (x = x + ...). This occurs before the regular self-attention and Hadamard-MLP steps.

Decode Pipeline Wiring

The needle/model/decode.py file contains the helper _engram_kv that builds the KV tuple for each forward pass, orchestrating the flow from index computation to block injection. This separation allows the model to pre-compute engram indices while maintaining clean abstractions between memory retrieval and consumption.

Technical Implementation Details

Index Computation with _engram_indices

The _engram_indices function transforms the token list into table indices based on configurable hyperparameters: engram_orders (the n-gram sizes), engram_heads (number of attention heads per order), and engram_slots (hash table size). This hashing mechanism compresses variable-length contexts into fixed-size addresses.

Table Lookup and Projection

Inside Engram.__call__, the implementation performs the following operations:

1. Table indexing: The tables are indexed via tables[jnp.arange(self.num_tables), indices]
2. Linear projection: Retrieved vectors pass through lightweight dense layers (key_proj, value_proj) to produce the final key and value embeddings
3. Tap convolution: Values undergo blending through the tap convolution (taps + _shift_right), which mixes information from neighboring positions to smooth local context

Working with Engram KV Memory: Code Examples

Basic Usage with Default Configuration

import needle

# Create a Needle agent with the built‑in engram KV memory (enabled by default)

agent = needle.Needle(tools=[...])          # tools are described with @needle.tool

response = agent.run("Summarize the last three tool calls.") 
print(response["results"])

Customizing Memory Capacity and Injection Sites

from needle import Needle, config

# Adjust the engram hyper‑parameters

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # sites where engram KV is injected

    engram_slots=16384,         # size of each hash table

    engram_orders=(2, 3, 4),   # n‑gram orders used for indexing

    engram_heads=4,             # number of heads per order

)

agent = Needle(tools=[...], config=custom_cfg)
print(agent.run("What was the last tool you called?")["results"])

Low-Level Manual Retrieval

import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Assume we have a token array `tokens` and a compiled model `mdl`

cfg = agent.config
orders, heads, _ = engram_geometry(cfg)
indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

# Retrieve the KV pair from the first engram site

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(orders) * heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

k, v = engram(indices, ngram_ok=jnp.ones(indices.shape[:-1]), tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), quant=False)

# `k` and `v` can now be passed to a Block as `engram_kv=(k, v)`

Memory Bounds and Performance Characteristics

The engram KV memory provides a persistent KV sink for tool-related tokens, enabling the model to remember tool calls across a long interaction without growing RAM usage. Because the tables are fixed-size and the sliding window is limited to 256 tokens, the memory cost remains constant at approximately 28 MiB regardless of conversation length. This bounded approach contrasts sharply with standard transformer KV caches that consume GPU memory proportional to sequence length multiplied by batch size and head dimensions.

Summary

• Engram KV memory uses fixed-size hash tables to store n-gram representations with a constant footprint of ~28 MiB
• The system fires at configured layers specified by engram_layers and integrates inside Block.__call__ (lines 18-27 of needle/model/architecture.py) via attention-style gating
• Index computation occurs via _engram_indices using engram_orders, engram_heads, and engram_slots parameters
• Retrieved values undergo tap convolution for position mixing before being projected through key_proj and value_proj layers
• This architecture enables long-horizon tool calling without the linear memory growth associated with traditional KV caches

Frequently Asked Questions

What is the fixed memory footprint of Engram KV memory in Needle 2?

The engram KV memory maintains a constant footprint of approximately 28 MiB regardless of conversation length. This bounded size results from fixed-size hash tables and a 256-token sliding window, as implemented in the Engram class (lines 81-106 of needle/model/architecture.py).

How does Engram KV memory differ from a standard transformer KV cache?

Standard transformer KV caches grow linearly with sequence length, consuming additional memory for every new token. Engram KV memory uses hashed n-gram indexing into fixed-size tables, providing persistent context storage with constant memory usage independent of sequence length.

Where exactly does Engram KV memory integrate with the transformer blocks?

The memory integrates inside Block.__call__ at lines 18-27 of needle/model/architecture.py, occurring before standard self-attention. It receives the engram_kv tuple computed by _engram_kv in decode.py, applies attention-style gating (alpha), and adds the gated contribution to the hidden state.

Can developers customize the n-gram hashing behavior for Engram memory?

Yes. Developers control the hashing via the engram_orders, engram_heads, and engram_slots configuration parameters. These settings determine which n-gram sizes are hashed, how many parallel heads process each order, and the size of the underlying hash tables, allowing trade-offs between collision resistance and memory consumption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}