What Is Engram Key-Value Memory in Needle 2? Architecture and Implementation Guide

Engram key-value memory in Needle 2 is a fixed-size, hashed n-gram memory system that stores recent token representations in compact lookup tables, enabling persistent tool-call recall without unbounded RAM growth.

Needle 2, developed by Cactus Compute, augments its attention mechanism with an innovative engram key-value (KV) memory designed specifically for bounded-memory inference. This architectural component addresses the challenge of maintaining context over long tool-using conversations by storing hashed n-gram representations in fixed-size tables rather than allowing memory usage to grow linearly with sequence length.

Core Architecture and Design Principles

The engram KV memory consists of fixed-size embedding tables that store vector slots indexed by n-gram hashes of the recent token stream. According to the implementation in needle/model/architecture.py, the Engram class defines these tables and manages the compression of historical context into a compact, sliding-window memory footprint of approximately 28 MiB.

Memory Tables and N-gram Hashing

At its foundation, the system uses configurable hyperparameters including engram_orders, engram_heads, and engram_slots to determine how tokens map to table indices. The _engram_indices function converts incoming token lists into table indices based on these n-gram orders, creating a sparse addressing scheme that captures local context patterns while maintaining constant memory usage.

Convolutional Value Refinement

Retrieved values undergo additional processing through a "tap" convolution mechanism. The implementation applies _shift_right operations combined with convolutional filters (taps) to blend neighboring positions, enriching the value representations with local positional context before integration into the main model flow.

Implementation Workflow in Needle 2

Index Computation

The process begins with _engram_indices, which processes the incoming token stream to generate indices for the memory tables. This function considers the configured n-gram orders and head count to create a multi-dimensional addressing scheme that maps token sequences to specific slots within the fixed-size tables defined by engram_slots.

Table Lookup and Projection

Inside Engram.__call__ (lines 81-106 of needle/model/architecture.py), the system performs table lookups using tables[jnp.arange(self.num_tables), indices] to retrieve raw embeddings. These embeddings pass through lightweight projection layers—key_proj and value_proj—to generate the final key and value vectors used in downstream attention mechanisms.

Block-Level Integration

Each Block in the architecture receives an optional engram_kv tuple containing the computed keys and values. As implemented in Block.__call__ (lines 18-27 of architecture.py), the block computes an attention-style gating coefficient (alpha) between the current hidden state and the retrieved engram values, then adds this gated contribution to the hidden representation before standard self-attention and Hadamard-MLP processing.

Configuration and Usage Examples

Needle 2 enables engram memory by default, but developers can customize its behavior through the configuration API or access it directly for advanced use cases.

Basic Usage with Default Configuration

import needle

# Create agent with built-in engram KV memory enabled by default

agent = needle.Needle(tools=[...])
response = agent.run("Summarize the last three tool calls.")
print(response["results"])

Customizing Memory Parameters

from needle import Needle, config

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # Injection sites where engram KV is integrated

    engram_slots=16384,         # Size of each hash table

    engram_orders=(2, 3, 4),    # N-gram orders used for indexing

    engram_heads=4,             # Number of heads per order

)

agent = Needle(tools=[...], config=custom_cfg)

Low-Level Manual Access

import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Compute indices from tokens using configured geometry

indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

# Initialize engram module

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(orders) * heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

# Retrieve KV pairs

k, v = engram(
    indices, 
    ngram_ok=jnp.ones(indices.shape[:-1]), 
    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), 
    quant=False
)

# Pass engram_kv=(k, v) to Block

Memory Efficiency and Practical Benefits

The engram KV memory provides a persistent KV sink specifically optimized for tool-related tokens. Unlike standard KV caches that grow linearly with conversation length, Needle 2's implementation maintains constant memory usage regardless of interaction duration by limiting the sliding window to 256 tokens and using fixed-size tables. This bounded approach, documented in the repository's README.md and exemplified in tests/conftest.py, ensures that long-running tool invocations do not exhaust available RAM or degrade inference performance.

Summary

• Engram key-value memory uses fixed-size hash tables to store n-gram representations of recent tokens, maintaining a bounded memory footprint of approximately 28 MiB regardless of conversation length.
• The system computes indices via _engram_indices, retrieves vectors through Engram.__call__, and integrates them into transformer blocks using attention-style gating in Block.__call__.
• Configuration options in needle/model/architecture.py and tests/conftest.py control table size (engram_slots), n-gram orders (engram_orders), and injection layers (engram_layers).
• This architecture enables persistent recall of tool calls across long conversations without the linear memory growth typical of standard transformer KV caches.

Frequently Asked Questions

How does engram key-value memory differ from standard KV caching?

Standard KV caches append key-value pairs for every new token, causing memory usage to grow linearly with sequence length. Engram KV memory uses fixed-size hash tables indexed by n-grams, maintaining constant memory footprint through a 256-token sliding window while still capturing recent context patterns through multi-order hashing.

What file contains the main Engram class implementation?

The core implementation resides in needle/model/architecture.py (lines 81-106), which defines the Engram class, index computation functions, and the integration logic within Block.__call__. Additional wiring appears in needle/model/decode.py through the _engram_kv helper function that builds the KV tuple for each forward pass.

Can I disable or customize the engram memory size?

Yes. While enabled by default, you can customize parameters through needle.config.Config by adjusting engram_slots (table size), engram_orders (n-gram lengths), engram_heads (parallel attention heads), and engram_layers (injection points). The tests/conftest.py file provides concrete examples of typical hyper-parameter configurations used in production deployments.

Why is the memory footprint limited to approximately 28 MiB?

The fixed table sizes and constrained sliding window ensure that engram storage does not scale with input length. This bounded memory design, explicitly documented in the README.md, specifically supports long-running tool interactions where unbounded memory growth would otherwise cause OOM errors or performance degradation during extended conversations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}