# Understanding Engram Layers in Needle 2: Architecture and Function

> Discover engram layers in Needle 2, enhancing transformers with hashed n-gram memory for efficient long-context retrieval. Learn their architecture and function.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-17

---

**Engram layers in Needle 2 are specific transformer layers that augment standard attention with a fixed-size hashed n-gram key-value memory, enabling efficient long-context retrieval without expanding the attention matrix.**

The Needle 2 repository introduces **engram layers** as a mechanism to enhance transformer performance on resource-constrained devices. These specialized layers integrate hashed n-gram tables directly into the attention mechanism, allowing the model to attend to recurring token patterns across long contexts while maintaining a compact computational footprint.

## What Are Engram Layers in Needle 2?

Engram layers are designated transformer layers that consult an external **engram** key-value memory during the forward pass. Unlike standard attention, which computes affinities across the full context window, engram layers retrieve fixed-size embeddings from hashed n-gram tables stored in `Engram` modules. According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the model creates one `Engram` instance for every layer index specified in the configuration, typically defaulting to layers `(2, 15)`【L71-L75】. When the forward pass reaches one of these layers, cached n-gram key-value pairs are injected, allowing the model to access associative memory without growing the quadratic attention matrix.

## Configuration and Geometry

The placement and structure of engram layers are governed by the `TransformerConfig` class and the `engram_geometry()` function.

**Layer Specification** – Developers define which layers contain engram memory via the `engram_layers` parameter. The default tuple `(2, 15)` places engram modules at layer indices 2 and 15 during model setup【L71-L75】.

**Dimensional Layout** – The `engram_geometry()` function computes the structural parameters required by each engram site, including the number of n-gram orders, the head count, and the per-head sub-dimension【L20-L24】. This geometry ensures that retrieved vectors align with the model’s internal head dimensions for seamless concatenation.

## The Hashing and Indexing Pipeline

To retrieve values from the engram tables, Needle 2 converts input token IDs into slot indices using the `engram_indices()` function.

The hashing process combines three elements:
- A **per-order seed** unique to each n-gram order
- A **shifted-right version** of the input token stream
- A **prime-multiplicative mix** followed by a modulo operation against the slot count

This computation occurs in `engram_indices()` at lines 46–57 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)【L46-L57】, producing deterministic slot indices for every order-head pair that map into the fixed-size embedding tables.

## Memory Retrieval and Attention Integration

During the forward pass, the `SimpleAttentionNetwork._engram_kv()` method orchestrates the retrieval and integration of engram memory.

**KV Retrieval** – The method passes computed token indices to each active `Engram` module. These modules store learnable `tables` of embeddings and project retrieved vectors into key and value tensors via `key_proj` and `value_proj` layers【L81-L100】.

**Mask Construction** – `_engram_kv()` also builds boolean masks named `ngram_ok` and `tap_ok` that enforce causal constraints, controlling which n-grams and convolutional taps are permitted given the autoregressive mask【L9-L18】.

**Concatenation** – The projected key-value pairs are concatenated with the standard attention KV pairs inside the model’s `Stack` operation, effectively augmenting the attention mechanism with associative memory at specific layer depths【L90-L95】.

## Working with Engram Layers in Code

You can configure and inspect engram layers using the `TransformerConfig` class and the `Needle` model interface.

```python
import needle
from needle.model.architecture import TransformerConfig

# 1️⃣ Create a model with default engram layers (2 and 15)

cfg = TransformerConfig()               # engram_layers = (2, 15) by default

model = needle.Needle(tools=[...], config=cfg)

```

To verify which layers contain engram memory:

```python

# 2️⃣ Inspect which layers have engrams

print("Engram layers:", cfg.engram_layers)   # → (2, 15)

```

You can observe engram KV usage during inference by enabling detailed return values:

```python

# 3️⃣ Run a query and see the engram KV being used (debug mode)

logits, hidden = model(tokens=needle.tokenize("What is the weather in Paris?"),
                       return_mtp=True)      # `return_mtp` forces the forward path

# The internal call to `_engram_kv` runs at layers 2 and 15

```

To customize the placement of engram layers, modify the configuration tuple:

```python

# 4️⃣ Customise engram layers (e.g., only at layer 5)

custom_cfg = TransformerConfig(engram_layers=(5,))
custom_model = needle.Needle(tools=[...], config=custom_cfg)
print("Engram layers:", custom_cfg.engram_layers)   # → (5,)

```

## Key Implementation Files

The engram layer functionality spans several modules:
- [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) – Contains `TransformerConfig`, `engram_geometry()`, `engram_indices()`, the `Engram` module, and `SimpleAttentionNetwork` integration logic【L20-L24】【L46-L57】【L71-L100】.
- [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) – Handles serialization of engram tables and projection weights.
- [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) – Defines naming conventions for engram parameters during quantization (e.g., mapping `engrams_0.embedding` to `engram0.tables`).

## Summary

- **Engram layers** are specific transformer layers (default: 2 and 15) that augment attention with hashed n-gram memory.
- The `engram_geometry()` function calculates dimensional requirements for n-gram orders and attention heads.
- `engram_indices()` hashes token IDs into table slots using per-order seeds and prime-based mixing.
- The `Engram` module stores embeddings in `tables` and projects them via `key_proj` and `value_proj` for concatenation with standard KV pairs.
- `SimpleAttentionNetwork._engram_kv()` retrieves values and applies causal masks (`ngram_ok`, `tap_ok`) at the specified layer indices.

## Frequently Asked Questions

### What are engram layers in Needle 2?

Engram layers are specific indices in the transformer stack where the model consults a fixed-size associative memory of hashed n-grams. They function as augmentations to standard attention, allowing the model to retrieve key-value pairs from learned tables rather than computing full attention over long contexts.

### How does the hashing mechanism work in engram layers?

The `engram_indices()` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) computes slot indices by combining a per-order random seed with a shifted-right token stream, applying a prime-multiplicative hash, and taking the modulo with the table size. This deterministic process maps variable-length contexts to fixed embedding slots.

### Where are engram layers configured?

Engram layers are defined in the `TransformerConfig` class via the `engram_layers` parameter, which accepts a tuple of integer layer indices. The default configuration uses layers `(2, 15)`, but this can be customized to place engram memory at any single or multiple layers in the stack.

### How do engram layers improve model performance?

By storing recurring token patterns in fixed-size tables and retrieving them via constant-time hashing, engram layers reduce the computational overhead of long-context modeling. This architecture is specifically optimized for tool-calling scenarios on tiny devices where full attention over extended contexts would be prohibitively expensive.