Understanding Engram Layers in Needle 2: Architecture and Function

Engram layers in Needle 2 are specific transformer layers that augment standard attention with a fixed-size hashed n-gram key-value memory, enabling efficient long-context retrieval without expanding the attention matrix.

The Needle 2 repository introduces engram layers as a mechanism to enhance transformer performance on resource-constrained devices. These specialized layers integrate hashed n-gram tables directly into the attention mechanism, allowing the model to attend to recurring token patterns across long contexts while maintaining a compact computational footprint.

What Are Engram Layers in Needle 2?

Engram layers are designated transformer layers that consult an external engram key-value memory during the forward pass. Unlike standard attention, which computes affinities across the full context window, engram layers retrieve fixed-size embeddings from hashed n-gram tables stored in Engram modules. According to the source code in needle/model/architecture.py, the model creates one Engram instance for every layer index specified in the configuration, typically defaulting to layers (2, 15)【L71-L75】. When the forward pass reaches one of these layers, cached n-gram key-value pairs are injected, allowing the model to access associative memory without growing the quadratic attention matrix.

Configuration and Geometry

The placement and structure of engram layers are governed by the TransformerConfig class and the engram_geometry() function.

Layer Specification – Developers define which layers contain engram memory via the engram_layers parameter. The default tuple (2, 15) places engram modules at layer indices 2 and 15 during model setup【L71-L75】.

Dimensional Layout – The engram_geometry() function computes the structural parameters required by each engram site, including the number of n-gram orders, the head count, and the per-head sub-dimension【L20-L24】. This geometry ensures that retrieved vectors align with the model’s internal head dimensions for seamless concatenation.

The Hashing and Indexing Pipeline

To retrieve values from the engram tables, Needle 2 converts input token IDs into slot indices using the engram_indices() function.

The hashing process combines three elements:

• A per-order seed unique to each n-gram order
• A shifted-right version of the input token stream
• A prime-multiplicative mix followed by a modulo operation against the slot count

This computation occurs in engram_indices() at lines 46–57 of needle/model/architecture.py【L46-L57】, producing deterministic slot indices for every order-head pair that map into the fixed-size embedding tables.

Memory Retrieval and Attention Integration

During the forward pass, the SimpleAttentionNetwork._engram_kv() method orchestrates the retrieval and integration of engram memory.

KV Retrieval – The method passes computed token indices to each active Engram module. These modules store learnable tables of embeddings and project retrieved vectors into key and value tensors via key_proj and value_proj layers【L81-L100】.

Mask Construction – _engram_kv() also builds boolean masks named ngram_ok and tap_ok that enforce causal constraints, controlling which n-grams and convolutional taps are permitted given the autoregressive mask【L9-L18】.

Concatenation – The projected key-value pairs are concatenated with the standard attention KV pairs inside the model’s Stack operation, effectively augmenting the attention mechanism with associative memory at specific layer depths【L90-L95】.

Working with Engram Layers in Code

You can configure and inspect engram layers using the TransformerConfig class and the Needle model interface.

import needle
from needle.model.architecture import TransformerConfig

# 1️⃣ Create a model with default engram layers (2 and 15)

cfg = TransformerConfig()               # engram_layers = (2, 15) by default

model = needle.Needle(tools=[...], config=cfg)

To verify which layers contain engram memory:

# 2️⃣ Inspect which layers have engrams

print("Engram layers:", cfg.engram_layers)   # → (2, 15)

You can observe engram KV usage during inference by enabling detailed return values:

# 3️⃣ Run a query and see the engram KV being used (debug mode)

logits, hidden = model(tokens=needle.tokenize("What is the weather in Paris?"),
                       return_mtp=True)      # `return_mtp` forces the forward path

# The internal call to `_engram_kv` runs at layers 2 and 15

To customize the placement of engram layers, modify the configuration tuple:

# 4️⃣ Customise engram layers (e.g., only at layer 5)

custom_cfg = TransformerConfig(engram_layers=(5,))
custom_model = needle.Needle(tools=[...], config=custom_cfg)
print("Engram layers:", custom_cfg.engram_layers)   # → (5,)

Key Implementation Files

The engram layer functionality spans several modules:

• needle/model/architecture.py – Contains TransformerConfig, engram_geometry(), engram_indices(), the Engram module, and SimpleAttentionNetwork integration logic【L20-L24】【L46-L57】【L71-L100】.
• needle/model/export.py – Handles serialization of engram tables and projection weights.
• needle/model/quantize.py – Defines naming conventions for engram parameters during quantization (e.g., mapping engrams_0.embedding to engram0.tables).

Summary

• Engram layers are specific transformer layers (default: 2 and 15) that augment attention with hashed n-gram memory.
• The engram_geometry() function calculates dimensional requirements for n-gram orders and attention heads.
• engram_indices() hashes token IDs into table slots using per-order seeds and prime-based mixing.
• The Engram module stores embeddings in tables and projects them via key_proj and value_proj for concatenation with standard KV pairs.
• SimpleAttentionNetwork._engram_kv() retrieves values and applies causal masks (ngram_ok, tap_ok) at the specified layer indices.

Frequently Asked Questions

What are engram layers in Needle 2?

Engram layers are specific indices in the transformer stack where the model consults a fixed-size associative memory of hashed n-grams. They function as augmentations to standard attention, allowing the model to retrieve key-value pairs from learned tables rather than computing full attention over long contexts.

How does the hashing mechanism work in engram layers?

The engram_indices() function in needle/model/architecture.py computes slot indices by combining a per-order random seed with a shifted-right token stream, applying a prime-multiplicative hash, and taking the modulo with the table size. This deterministic process maps variable-length contexts to fixed embedding slots.

Where are engram layers configured?

Engram layers are defined in the TransformerConfig class via the engram_layers parameter, which accepts a tuple of integer layer indices. The default configuration uses layers (2, 15), but this can be customized to place engram memory at any single or multiple layers in the stack.

How do engram layers improve model performance?

By storing recurring token patterns in fixed-size tables and retrieving them via constant-time hashing, engram layers reduce the computational overhead of long-context modeling. This architecture is specifically optimized for tool-calling scenarios on tiny devices where full attention over extended contexts would be prohibitively expensive.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}