# What Is Engram Key-Value Memory in Needle 2? Architecture and Implementation Guide

> Explore Engram key-value memory in Needle 2. Learn how this fixed-size n-gram system efficiently stores token representations for persistent tool-call recall without RAM issues.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-21

---

**Engram key-value memory in Needle 2 is a fixed-size, hashed n-gram memory system that stores recent token representations in compact lookup tables, enabling persistent tool-call recall without unbounded RAM growth.**

Needle 2, developed by Cactus Compute, augments its attention mechanism with an innovative **engram key-value (KV) memory** designed specifically for bounded-memory inference. This architectural component addresses the challenge of maintaining context over long tool-using conversations by storing hashed n-gram representations in fixed-size tables rather than allowing memory usage to grow linearly with sequence length.

## Core Architecture and Design Principles

The engram KV memory consists of fixed-size embedding tables that store vector slots indexed by n-gram hashes of the recent token stream. According to the implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `Engram` class defines these tables and manages the compression of historical context into a compact, sliding-window memory footprint of approximately 28 MiB.

### Memory Tables and N-gram Hashing

At its foundation, the system uses configurable hyperparameters including `engram_orders`, `engram_heads`, and `engram_slots` to determine how tokens map to table indices. The `_engram_indices` function converts incoming token lists into table indices based on these n-gram orders, creating a sparse addressing scheme that captures local context patterns while maintaining constant memory usage.

### Convolutional Value Refinement

Retrieved values undergo additional processing through a "tap" convolution mechanism. The implementation applies `_shift_right` operations combined with convolutional filters (`taps`) to blend neighboring positions, enriching the value representations with local positional context before integration into the main model flow.

## Implementation Workflow in Needle 2

### Index Computation

The process begins with `_engram_indices`, which processes the incoming token stream to generate indices for the memory tables. This function considers the configured n-gram orders and head count to create a multi-dimensional addressing scheme that maps token sequences to specific slots within the fixed-size tables defined by `engram_slots`.

### Table Lookup and Projection

Inside `Engram.__call__` (lines 81-106 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)), the system performs table lookups using `tables[jnp.arange(self.num_tables), indices]` to retrieve raw embeddings. These embeddings pass through lightweight projection layers—`key_proj` and `value_proj`—to generate the final key and value vectors used in downstream attention mechanisms.

### Block-Level Integration

Each `Block` in the architecture receives an optional `engram_kv` tuple containing the computed keys and values. As implemented in `Block.__call__` (lines 18-27 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)), the block computes an attention-style gating coefficient (`alpha`) between the current hidden state and the retrieved engram values, then adds this gated contribution to the hidden representation before standard self-attention and Hadamard-MLP processing.

## Configuration and Usage Examples

Needle 2 enables engram memory by default, but developers can customize its behavior through the configuration API or access it directly for advanced use cases.

### Basic Usage with Default Configuration

```python
import needle

# Create agent with built-in engram KV memory enabled by default

agent = needle.Needle(tools=[...])
response = agent.run("Summarize the last three tool calls.")
print(response["results"])

```

### Customizing Memory Parameters

```python
from needle import Needle, config

custom_cfg = config.Config(
    engram_layers=(2, 8, 14),   # Injection sites where engram KV is integrated

    engram_slots=16384,         # Size of each hash table

    engram_orders=(2, 3, 4),    # N-gram orders used for indexing

    engram_heads=4,             # Number of heads per order

)

agent = Needle(tools=[...], config=custom_cfg)

```

### Low-Level Manual Access

```python
import jax.numpy as jnp
from needle.model.architecture import engram_indices, Engram

# Compute indices from tokens using configured geometry

indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

# Initialize engram module

engram = Engram(
    d_model=cfg.d_model,
    num_tables=len(orders) * heads,
    slots=cfg.engram_slots,
    sub_dim=ENGRAM_SUB_DIM,
    num_layers=cfg.num_layers,
    conv_dilation=1,
)

# Retrieve KV pairs

k, v = engram(
    indices, 
    ngram_ok=jnp.ones(indices.shape[:-1]), 
    tap_ok=jnp.ones((ENGRAM_CONV_TAPS,)), 
    quant=False
)

# Pass engram_kv=(k, v) to Block

```

## Memory Efficiency and Practical Benefits

The engram KV memory provides a **persistent KV sink** specifically optimized for tool-related tokens. Unlike standard KV caches that grow linearly with conversation length, Needle 2's implementation maintains constant memory usage regardless of interaction duration by limiting the sliding window to 256 tokens and using fixed-size tables. This bounded approach, documented in the repository's [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) and exemplified in [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py), ensures that long-running tool invocations do not exhaust available RAM or degrade inference performance.

## Summary

- **Engram key-value memory** uses fixed-size hash tables to store n-gram representations of recent tokens, maintaining a bounded memory footprint of approximately 28 MiB regardless of conversation length.
- The system computes indices via `_engram_indices`, retrieves vectors through `Engram.__call__`, and integrates them into transformer blocks using attention-style gating in `Block.__call__`.
- Configuration options in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py) control table size (`engram_slots`), n-gram orders (`engram_orders`), and injection layers (`engram_layers`).
- This architecture enables persistent recall of tool calls across long conversations without the linear memory growth typical of standard transformer KV caches.

## Frequently Asked Questions

### How does engram key-value memory differ from standard KV caching?

Standard KV caches append key-value pairs for every new token, causing memory usage to grow linearly with sequence length. **Engram KV memory** uses fixed-size hash tables indexed by n-grams, maintaining constant memory footprint through a 256-token sliding window while still capturing recent context patterns through multi-order hashing.

### What file contains the main Engram class implementation?

The core implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 81-106), which defines the `Engram` class, index computation functions, and the integration logic within `Block.__call__`. Additional wiring appears in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) through the `_engram_kv` helper function that builds the KV tuple for each forward pass.

### Can I disable or customize the engram memory size?

Yes. While enabled by default, you can customize parameters through `needle.config.Config` by adjusting `engram_slots` (table size), `engram_orders` (n-gram lengths), `engram_heads` (parallel attention heads), and `engram_layers` (injection points). The [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py) file provides concrete examples of typical hyper-parameter configurations used in production deployments.

### Why is the memory footprint limited to approximately 28 MiB?

The fixed table sizes and constrained sliding window ensure that engram storage does not scale with input length. This **bounded memory** design, explicitly documented in the [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md), specifically supports long-running tool interactions where unbounded memory growth would otherwise cause OOM errors or performance degradation during extended conversations.