How the Engram Memory Retrieval System Avoids Full Attention Contexts in Needle

The Needle model eliminates quadratic attention complexity by storing compressed key/value pairs in fixed-size external memory tables, retrieving context via constant-time hash lookups, and selectively integrating memory only at designated layers rather than attending over the full token sequence.

The cactus-compute/needle repository implements a novel engram memory retrieval system that augments standard Transformer architectures with external, compressed memory tables. Instead of computing attention over every prior token—which scales as O(N²) with sequence length—Needle retrieves context through lightweight hash-based indexing into static memory slots. This design maintains long-range recall capabilities while dramatically reducing memory bandwidth and computational requirements.

External Memory Tables Replace Full-Sequence Context

The architecture creates distinct memory tables for each engram site, defined in needle/model/architecture.py at lines 81-95. These tables store parameters of shape (num_tables, slots, sub_dim), populated during training and remaining static during inference.

Unlike standard attention mechanisms that must load the entire key/value cache, the engram tables exist as discrete parameters:

tables = self.param("embedding", default_init(),
                    (self.num_tables, self.slots, self.sub_dim))

This separation allows the model to access long-range information without maintaining a full history of token activations in working memory. According to the Needle source code, the table remains immutable during inference, acting as a compressed knowledge base separate from the dynamic attention cache.

Hash-Based Indexing for Constant-Time Lookup

To avoid scanning the entire memory table, Needle uses the engram_indices function (lines 46-57 in architecture.py) to hash tokens directly into memory slots. This reversible operation mixes token IDs across configured order and head dimensions, producing a compact integer index in the range [0, slots).

The hashing mechanism operates independently of sequence length, ensuring that locating the relevant memory slot requires constant time regardless of how many tokens precede the current position. As implemented in the Needle repository, this avoids the linear scanning that would otherwise reintroduce O(N) complexity per token.

Sparse Retrieval from Fixed Memory Slots

Using the computed indices, the model fetches only the specific rows corresponding to the current token window. The retrieval operation performs a targeted lookup rather than dense matrix multiplication:

fetched = tables[jnp.arange(self.num_tables), indices]  # → (B, T, num_tables, sub_dim)

This sparse retrieval mechanism, found at lines 81-106 in architecture.py, eliminates the need for inner-product calculations over the full token history. The advanced indexing operation selects exactly num_tables entries per token, independent of sequence length.

For direct retrieval outside the standard forward pass, the repository exposes the geometry and indexing utilities:

from needle.model.architecture import engram_indices, _mask_diag

# Compute hash indices from token IDs

orders, heads, _ = model.engram_geometry(cfg)
indices = engram_indices(tokens, orders, heads, cfg.engram_slots)

# Build causal masks for n-gram and tap contexts

ngram_ok = jnp.stack([_mask_diag(mask, o - 1) for o in orders for _ in range(heads)], axis=-1)
tap_ok = jnp.stack([_mask_diag(mask, j * max(orders)) for j in range(4)], axis=0)

# Pull memory for each site

kv = [e(indices, ngram_ok, tap_ok) for e in model.engrams]
keys, values = jnp.stack([k for k, _ in kv]), jnp.stack([v for _, v in kv])

This approach retrieves only the pre-computed engram values without materializing the full attention matrix.

Convolutional Taps for Local Temporal Aggregation

After retrieval, the engram values undergo processing by a 1-D convolution with ENGRAM_CONV_TAPS = 4 (implemented at lines 200-206). These "taps" aggregate information from only the four most recent positions:

# Inside Engram.__call__ - convolution over limited history

This convolutional step, visible in the Engram class implementation, provides local context smoothing without reintroducing global attention dependencies. By limiting the receptive field to four positions, the system ensures that processing remains strictly local after the constant-time global lookup.

Selective Integration at Engram Layers

The retrieved key/value pairs (engram_kv) integrate with the main attention path only at specific engram layers—by default, layers 2 and 15—controlled by site_flags within Block.__call__ (lines 18-26 and 191-210).

The mechanism computes a similarity gate (alpha) between the current hidden state and retrieved memory:

# Inside Block.__call__ (lines 191-210)

alpha = nn.sigmoid(jnp.einsum("btd,sbtd->sbt", _rms_unit(x), _rms_unit(ek)) / math.sqrt(self.d_model))
x = x + jnp.einsum("s,sbt,sbtd->btd", site_flags, alpha, ev)

The site_flags tensor ensures that only the designated engram layers receive memory contributions, while alpha scales the integration based on relevance. This selective approach means the majority of layers process only standard self-attention over local windows, not the full historical context.

You can configure which layers participate via TransformerConfig:

from needle.model.architecture import TransformerConfig

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    engram_layers=(2, 15),   # Only these layers query engram memory

    engram_slots=8192,
    engram_orders=(2, 3),
)

Budget-Based KV Window Constraints

The model enforces hard limits on context size through kv_budget_window (lines 103-112 in architecture.py). This calculation ensures that the key/value cache never exceeds hardware-friendly bounds, guaranteeing that memory usage remains predictable and bounded even during long sequence generation.

When combined with the static engram tables, this budget mechanism ensures that the engram memory retrieval system operates within fixed resource constraints regardless of input length.

Summary

• External Tables: Engram memory stores compressed representations in fixed-size tables (num_tables, slots, sub_dim) at lines 81-95, separate from the growing attention cache.
• Hash Indexing: The engram_indices function (lines 46-57) computes constant-time lookups, avoiding linear scans of token history.
• Sparse Retrieval: Only specific slots are fetched via tables[jnp.arange(self.num_tables), indices], eliminating dense matrix operations over the full sequence.
• Local Processing: Convolutional taps (ENGRAM_CONV_TAPS = 4 at lines 200-206) aggregate just four recent positions after retrieval.
• Layer Gating: The site_flags mechanism (lines 18-26) restricts engram integration to specific layers, preventing full-context attention in standard layers.
• Memory Budgets: kv_budget_window (lines 103-112) enforces strict caps on cache size for hardware efficiency.

Frequently Asked Questions

What is an engram memory retrieval system in machine learning?

An engram memory retrieval system is an external memory mechanism that stores compressed key/value representations in fixed-size lookup tables. Unlike standard attention, which computes similarity across the entire sequence, an engram system retrieves context through hash-based indexing, enabling constant-time access to learned historical information without the quadratic scaling costs of full self-attention.

How does Needle's engram system avoid O(N²) attention costs?

Needle avoids quadratic complexity through three mechanisms: (1) hash-based indexing via engram_indices for O(1) memory location lookups, (2) sparse retrieval that fetches only specific table rows using tables[jnp.arange(self.num_tables), indices] rather than computing over all tokens, and (3) selective integration that applies engram memory only at designated layers (controlled by site_flags in Block.__call__), while other layers use limited local attention.

At which layers does the engram memory integrate with the Transformer?

By default, engram memory integrates only at layers 2 and 15, as specified by the engram_layers parameter in TransformerConfig (lines 78-96 in SimpleAttentionNetwork.setup). The site_flags tensor in Block.__call__ (lines 18-26) uses this configuration to gate memory contributions, ensuring that the remaining layers process only standard local attention without engram overhead.

What limits the capacity of the engram memory in Needle?

Capacity is constrained by two factors: the fixed table dimensions (num_tables, slots, sub_dim) defined in the Engram class (lines 81-95), and the kv_budget_window calculation (lines 103-112) that caps the dynamic key/value cache size. The engram_slots parameter (defaulting to 8192) determines the hash table size, while the budget window ensures hardware-friendly memory bounds during inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}