Engram Key-Value Memory in Needle 2 Architecture: Technical Deep Dive

Engram key-value memory in Needle 2 is a fixed-size, hashed n-gram storage mechanism that augments the attention layer with a bounded persistent memory for tool-related tokens, consuming approximately 28 MiB regardless of conversation length.

The Needle 2 open-source agent framework introduces a novel memory architecture to handle long-running tool interactions without unbounded RAM growth. At the core of this design sits the engram key-value memory, a compact, sliding-window storage system that hashes recent token n-grams into fixed-size lookup tables. Unlike standard KV caches that grow linearly with sequence length, this mechanism maintains constant memory usage while providing persistent context for function calls across extended interactions.

What Is Engram Key-Value Memory?

The engram key-value memory consists of fixed-size embedding tables that store vector representations of recent token contexts. For every incoming token, the system derives indices from the token's n-gram context using the engram_indices function, addressing slots within hash tables to produce key (k) and value (v) vectors. These values undergo further refinement through a lightweight convolution operation ("tap") that mixes neighboring positions, creating a localized, compressed memory of the recent token stream.

According to the README.md and the implementation in needle/model/architecture.py, this design caps memory consumption at approximately 28 MiB. The bounded nature stems from fixed table sizes and a sliding window limited to 256 tokens, ensuring that RAM usage remains constant even during extended multi-turn conversations with numerous tool invocations.

How Engram KV Memory Works

The implementation spans three distinct phases: index computation, table retrieval with value refinement, and block-level integration.

Index Computation

The _engram_indices function transforms incoming token sequences into table indices based on configurable hyperparameters. These include engram_orders (defining n-gram sizes), engram_heads (parallel hashing functions per order), and engram_slots (table size). By hashing n-gram contexts, the system maps variable-length token histories to fixed-size table addresses, enabling constant-time lookups regardless of conversation depth.

Table Lookup and Value Refinement

Within the Engram class defined in needle/model/architecture.py, the __call__ method performs the actual memory retrieval. It indexes the embedding tables using tables[jnp.arange(self.num_tables), indices], then projects the retrieved vectors into keys and values via dedicated key_proj and value_proj layers. The values receive additional processing through the tap convolution—implemented using taps and _shift_right operations—which blends spatially adjacent memory slots to capture local context patterns.

Integration with Transformer Blocks

Each transformer Block optionally accepts an engram_kv tuple containing the retrieved key-value pairs. Inside Block.__call__ (lines 18-27 in needle/model/architecture.py), the block computes an attention-style gating parameter (alpha) that modulates the interaction between the current hidden state and the engram values. The gated contribution adds directly to the block's hidden representation (x = x + ...) before the standard self-attention and Hadamard-MLP processing steps.

Configuration and Usage

Needle 2 enables engram KV memory by default, but developers can customize or access it programmatically through the configuration API and low-level module interfaces.

Basic usage with default settings:

import needle

# Create agent with built-in engram KV memory (enabled by default)

agent = needle.Needle(tools=[...])
response = agent.run("Summarize the last three tool calls.")
print(response["results"])

Customizing memory parameters:

from needle import Needle, config

# Adjust engram hyperparameters for specific workloads

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # Injection sites

    engram_slots=16384,         # Hash table size

    engram_orders=(2, 3, 4),    # N-gram orders

    engram_heads=4,             # Heads per order

)

agent = Needle(tools=[...], config=custom_cfg)

Low-level manual retrieval:

import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Assuming `tokens` array and compiled model

cfg = agent.config
indices = engram_indices(tokens, cfg.engram_orders, cfg.engram_heads, cfg.engram_slots)

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(cfg.engram_orders) * cfg.engram_heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

k, v = engram(
    indices, 
    ngram_ok=jnp.ones(indices.shape[:-1]), 
    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), 
    quant=False
)

# Pass engram_kv=(k, v) to Block

Summary

• Engram key-value memory provides a fixed-size, hashed n-gram storage mechanism in Needle 2, bounding memory usage at approximately 28 MiB regardless of sequence length.
• The system uses _engram_indices to hash token contexts into table addresses based on configurable orders, heads, and slots defined in tests/conftest.py.
• Table lookups occur in Engram.__call__ with value refinement via tap convolution before injection into transformer blocks.
• Each Block integrates engram KV through attention-style gating (alpha) that augments hidden states before standard self-attention processing.
• The wiring into the model pipeline is handled by needle/model/decode.py, which constructs the KV tuple for each forward pass.

Frequently Asked Questions

What is the memory footprint of engram KV memory?

The engram key-value memory consumes approximately 28 MiB of RAM regardless of conversation length. This bounded footprint results from fixed-size hash tables and a sliding window limited to 256 tokens, ensuring constant memory usage during long-running agent interactions.

How does engram memory differ from standard KV caching?

Standard KV caches grow linearly with sequence length, storing every token's key and value vectors. In contrast, engram memory uses hashed n-gram indices to compress historical context into fixed-size tables, providing a persistent memory sink without unbounded growth, specifically optimized for retaining tool call context.

Can engram memory be disabled or customized?

Yes. While enabled by default, you can customize engram parameters through config.Config by adjusting engram_layers, engram_slots, engram_orders, and engram_heads. The low-level API in needle/model/architecture.py also allows manual control over engram computation and injection.

Which layers support engram KV injection?

Engram KV memory can be injected at arbitrary transformer layers specified by the engram_layers tuple (typically layers 2, 8, and 14 in default configurations). Each specified Block checks for the engram_kv parameter and applies the gated integration when present.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}