What Are Engram Layers in Needle 2? Architecture and Purpose Explained

Engram layers in Needle 2 are specialized Transformer blocks that incorporate a hash-based external memory system, enabling the model to retrieve long-range context in constant time without incurring the quadratic computational cost of standard self-attention.

Needle 2, developed in the cactus-compute/needle repository, augments the classic Transformer architecture with Engram modules that function as fast, learned memory caches. These layers store n-gram hashed token representations in embedding tables and blend them into the hidden state before attention computation. By wiring this external memory into specific blocks, Needle 2 achieves efficient long-sequence modeling while maintaining the parallel processing benefits of the base architecture.

What Are Engram Layers in Needle 2?

Engram layers are specific Transformer blocks identified by the engram_layers configuration tuple where an Engram sub-module is instantiated. Unlike standard blocks that rely solely on self-attention, these layers inject a hash-based retrieval mechanism that provides random access to historical token information stored in learned tables.

Core Architecture and Components

At the heart of each Engram layer lies the Engram class defined in needle/model/architecture.py. The model constructor instantiates these modules for each specified layer:

self.engrams = [
    Engram(cfg.d_model, len(orders) * heads, cfg.engram_slots,
           cfg.engram_sub_dim, cfg.num_engram_tables,
           cfg.engram_conv_taps, cfg.engram_conv_dilation)
    for _ in cfg.engram_layers
]

Each Engram module contains three critical components:

• Learned embedding tables: Hash-mapped storage indexed by n-gram hashed token indices, capable of holding up to engram_slots entries (default 8192).
• Key/value projections: Linear layers that transform retrieved embeddings into key and value vectors compatible with the Transformer's d_model dimension.
• Convolutional taps: Learned convolutional filters applied to retrieved values to enforce temporal smoothness and local coherence in the external memory reads.

How Engram Layers Differ from Standard Transformer Blocks

Standard Transformer blocks compute context through multi-head self-attention with O(n²) complexity relative to sequence length. Engram layers supplement this by maintaining external memory that provides O(1) lookup access to relevant past tokens. Rather than attending to all previous positions, the model hashes current tokens and retrieves cached representations from constant-time hash table lookups.

Purpose and Benefits of Engram Layers

The primary purpose of Engram layers is to decouple long-range context modeling from the computational constraints of dense attention matrices. This architecture serves four specific functional requirements.

Long-Range Context Without Quadratic Cost

By storing n-gram-hashed token indices across many past tokens, Engram layers enable retrieval of relevant historical information without expanding the attention window. The hash-to-slot mapping operates in constant time per token, reducing the memory pressure typically associated with long sequences in standard Transformers.

Cache-Like Retrieval Mechanism

The Engram system functions as a differentiable cache. During the forward pass, the _engram_kv function in needle/model/decode.py computes hashed indices from input tokens and retrieves corresponding embeddings from parallel hash tables. The taps parameters then apply a learned convolution to these values, ensuring that retrieved context is smoothed before integration.

Enhanced Attention Through External Memory

Before the self-attention computation, Engram layers fuse retrieved key/value pairs into the block's hidden state via a learned gating mechanism. As implemented in needle/model/architecture.py (lines 319-325), the blending occurs through:

x = x + jnp.einsum("s,sbt,sbtd->btd", site_flags, alpha, ev)

Here, alpha represents a learned gate that controls the contribution of Engram values (ev) to the main hidden state (x). This allows the model to condition its subsequent attention computation on both the current sequence and relevant historical context.

Implementation Details in the Needle 2 Codebase

The Engram functionality spans four critical files in the cactus-compute/needle repository, with clear separation between configuration, instantiation, runtime execution, and export functionality.

Model Configuration Parameters

The Config dataclass in needle/model/architecture.py exposes six parameters controlling Engram behavior:

• engram_layers: Tuple specifying which block indices receive Engram modules (default (2, 15)).
• engram_orders: N-gram lengths for the hashing function (e.g., (2, 3) for bigrams and trigrams).
• engram_heads: Number of parallel hash tables, calculated as d_model / (|orders| × engram_sub_dim).
• engram_slots: Size of each hash table (default 8192).
• engram_conv_taps: Kernel size for the convolution applied to retrieved values.
• engram_conv_dilation: Dilation factor spacing between convolution samples.

Test configurations in tests/conftest.py demonstrate alternative layer placements (e.g., engram_layers=(1,)), while needle/model/export.py handles serialization of Engram parameters for deployment.

Forward Pass Integration

During inference, each Block.__call__ accepts an optional engram_kv argument containing the Engram-produced key/value pair. When present, the block computes per-site alpha gates that determine how much the external memory influences the current hidden state. This selective gating allows the model to ignore irrelevant cached context when processing local sequences.

Configuring Engram Layers in Your Model

To instantiate a Needle 2 model with Engram layers, specify the target block indices and memory parameters in the configuration:

from needle.model.architecture import Config, Model

cfg = Config(
    d_model=512,
    engram_layers=(2, 15),   # Insert Engram modules at blocks 2 and 15

    engram_orders=(2, 3),    # Use 2-gram and 3-gram hashing

    engram_slots=8192,       # External memory capacity per table

    engram_conv_taps=4,      # Convolution window size

    engram_conv_dilation=2,  # Spacing between convolution samples

)

model = Model(cfg)

# During inference, Engram memory is automatically utilized

tokens = tokenizer.encode("Your long input sequence here...")
logits = model(tokens)

For debugging or analysis, access the raw Engram KV cache directly:

engram_kv = model._engram_kv(tokens, mask=None, quant=False)
keys, values = engram_kv
print(f"Engram keys shape: {keys.shape}")
print(f"Engram values shape: {values.shape}")

Summary

• Engram layers are specialized Transformer blocks in Needle 2 that integrate hash-based external memory at specific model depths configurable via engram_layers.
• They store n-gram hashed representations in learned embedding tables, enabling O(1) retrieval of long-range context without quadratic attention costs.
• Retrieved values are blended into hidden states via learned alpha gates before the self-attention step, enriching available context vectors.
• Key configuration parameters include engram_slots (memory capacity), engram_orders (hashing granularity), and engram_conv_taps (smoothing convolution).
• The implementation spans needle/model/architecture.py, needle/model/decode.py, and needle/model/export.py, providing a complete pipeline from configuration to deployment.

Frequently Asked Questions

How do Engram layers reduce computational complexity?

Engram layers replace the O(n²) attention computation for long-range dependencies with O(1) hash table lookups. By storing compressed n-gram representations in fixed-size slots (defaulting to 8192 entries), the model retrieves relevant historical context without computing attention scores over the entire sequence history, reducing complexity from quadratic to linear with respect to sequence length.

What is the difference between Engram slots and standard attention windows?

Standard attention windows restrict context to a fixed number of recent tokens or local neighborhoods, whereas Engram slots provide random access to learned representations of any previous position. The slots store hashed embeddings rather than raw token values, allowing retrieval of semantically similar patterns rather than requiring exact positional matches within a sliding window.

Can I use Engram layers with any Transformer architecture?

Engram layers are specific to the Needle 2 implementation in cactus-compute/needle. However, the concept requires only three adaptations for other architectures: implementing the Engram module with its embedding tables and convolutional taps, modifying the block forward pass to accept engram_kv parameters, and inserting the gating logic shown in needle/model/architecture.py lines 319-325.

How do I tune the engram_conv_taps and engram_conv_dilation parameters?

The engram_conv_taps parameter controls the receptive field of the convolution applied to retrieved values, while engram_conv_dilation determines the spacing between taps. Start with the defaults (4 taps, dilation 2) and increase taps if your sequences require smoother temporal blending of retrieved context, or adjust dilation to capture patterns at different time scales without increasing the parameter count of the convolutional filter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}