# What Are Engram Layers in Needle 2? Architecture and Purpose Explained

> Discover Engram layers in Needle 2. These Transformer blocks use external memory for constant time long-range context retrieval, avoiding quadratic costs.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-30

---

**Engram layers in Needle 2 are specialized Transformer blocks that incorporate a hash-based external memory system, enabling the model to retrieve long-range context in constant time without incurring the quadratic computational cost of standard self-attention.**

Needle 2, developed in the cactus-compute/needle repository, augments the classic Transformer architecture with *Engram* modules that function as fast, learned memory caches. These layers store n-gram hashed token representations in embedding tables and blend them into the hidden state before attention computation. By wiring this external memory into specific blocks, Needle 2 achieves efficient long-sequence modeling while maintaining the parallel processing benefits of the base architecture.

## What Are Engram Layers in Needle 2?

Engram layers are specific Transformer blocks identified by the `engram_layers` configuration tuple where an `Engram` sub-module is instantiated. Unlike standard blocks that rely solely on self-attention, these layers inject a hash-based retrieval mechanism that provides random access to historical token information stored in learned tables.

### Core Architecture and Components

At the heart of each Engram layer lies the `Engram` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). The model constructor instantiates these modules for each specified layer:

```python
self.engrams = [
    Engram(cfg.d_model, len(orders) * heads, cfg.engram_slots,
           cfg.engram_sub_dim, cfg.num_engram_tables,
           cfg.engram_conv_taps, cfg.engram_conv_dilation)
    for _ in cfg.engram_layers
]

```

Each `Engram` module contains three critical components:

- **Learned embedding tables**: Hash-mapped storage indexed by n-gram hashed token indices, capable of holding up to `engram_slots` entries (default 8192).
- **Key/value projections**: Linear layers that transform retrieved embeddings into key and value vectors compatible with the Transformer's `d_model` dimension.
- **Convolutional taps**: Learned convolutional filters applied to retrieved values to enforce temporal smoothness and local coherence in the external memory reads.

### How Engram Layers Differ from Standard Transformer Blocks

Standard Transformer blocks compute context through multi-head self-attention with O(n²) complexity relative to sequence length. Engram layers supplement this by maintaining external memory that provides O(1) lookup access to relevant past tokens. Rather than attending to all previous positions, the model hashes current tokens and retrieves cached representations from constant-time hash table lookups.

## Purpose and Benefits of Engram Layers

The primary purpose of Engram layers is to decouple long-range context modeling from the computational constraints of dense attention matrices. This architecture serves four specific functional requirements.

### Long-Range Context Without Quadratic Cost

By storing n-gram-hashed token indices across many past tokens, Engram layers enable retrieval of relevant historical information without expanding the attention window. The hash-to-slot mapping operates in constant time per token, reducing the memory pressure typically associated with long sequences in standard Transformers.

### Cache-Like Retrieval Mechanism

The Engram system functions as a differentiable cache. During the forward pass, the `_engram_kv` function in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) computes hashed indices from input tokens and retrieves corresponding embeddings from parallel hash tables. The `taps` parameters then apply a learned convolution to these values, ensuring that retrieved context is smoothed before integration.

### Enhanced Attention Through External Memory

Before the self-attention computation, Engram layers fuse retrieved key/value pairs into the block's hidden state via a learned gating mechanism. As implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 319-325), the blending occurs through:

```python
x = x + jnp.einsum("s,sbt,sbtd->btd", site_flags, alpha, ev)

```

Here, `alpha` represents a learned gate that controls the contribution of Engram values (`ev`) to the main hidden state (`x`). This allows the model to condition its subsequent attention computation on both the current sequence and relevant historical context.

## Implementation Details in the Needle 2 Codebase

The Engram functionality spans four critical files in the cactus-compute/needle repository, with clear separation between configuration, instantiation, runtime execution, and export functionality.

### Model Configuration Parameters

The `Config` dataclass in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) exposes six parameters controlling Engram behavior:

- **`engram_layers`**: Tuple specifying which block indices receive Engram modules (default `(2, 15)`).
- **`engram_orders`**: N-gram lengths for the hashing function (e.g., `(2, 3)` for bigrams and trigrams).
- **`engram_heads`**: Number of parallel hash tables, calculated as `d_model / (|orders| × engram_sub_dim)`.
- **`engram_slots`**: Size of each hash table (default 8192).
- **`engram_conv_taps`**: Kernel size for the convolution applied to retrieved values.
- **`engram_conv_dilation`**: Dilation factor spacing between convolution samples.

Test configurations in [`tests/conftest.py`](https://github.com/cactus-compute/needle/blob/main/tests/conftest.py) demonstrate alternative layer placements (e.g., `engram_layers=(1,)`), while [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) handles serialization of Engram parameters for deployment.

### Forward Pass Integration

During inference, each `Block.__call__` accepts an optional `engram_kv` argument containing the Engram-produced key/value pair. When present, the block computes per-site alpha gates that determine how much the external memory influences the current hidden state. This selective gating allows the model to ignore irrelevant cached context when processing local sequences.

## Configuring Engram Layers in Your Model

To instantiate a Needle 2 model with Engram layers, specify the target block indices and memory parameters in the configuration:

```python
from needle.model.architecture import Config, Model

cfg = Config(
    d_model=512,
    engram_layers=(2, 15),   # Insert Engram modules at blocks 2 and 15

    engram_orders=(2, 3),    # Use 2-gram and 3-gram hashing

    engram_slots=8192,       # External memory capacity per table

    engram_conv_taps=4,      # Convolution window size

    engram_conv_dilation=2,  # Spacing between convolution samples

)

model = Model(cfg)

# During inference, Engram memory is automatically utilized

tokens = tokenizer.encode("Your long input sequence here...")
logits = model(tokens)

```

For debugging or analysis, access the raw Engram KV cache directly:

```python
engram_kv = model._engram_kv(tokens, mask=None, quant=False)
keys, values = engram_kv
print(f"Engram keys shape: {keys.shape}")
print(f"Engram values shape: {values.shape}")

```

## Summary

- **Engram layers** are specialized Transformer blocks in Needle 2 that integrate hash-based external memory at specific model depths configurable via `engram_layers`.
- They store **n-gram hashed representations** in learned embedding tables, enabling O(1) retrieval of long-range context without quadratic attention costs.
- Retrieved values are **blended into hidden states** via learned alpha gates before the self-attention step, enriching available context vectors.
- Key configuration parameters include `engram_slots` (memory capacity), `engram_orders` (hashing granularity), and `engram_conv_taps` (smoothing convolution).
- The implementation spans [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py), and [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), providing a complete pipeline from configuration to deployment.

## Frequently Asked Questions

### How do Engram layers reduce computational complexity?

Engram layers replace the O(n²) attention computation for long-range dependencies with O(1) hash table lookups. By storing compressed n-gram representations in fixed-size slots (defaulting to 8192 entries), the model retrieves relevant historical context without computing attention scores over the entire sequence history, reducing complexity from quadratic to linear with respect to sequence length.

### What is the difference between Engram slots and standard attention windows?

Standard attention windows restrict context to a fixed number of recent tokens or local neighborhoods, whereas Engram slots provide random access to learned representations of any previous position. The slots store hashed embeddings rather than raw token values, allowing retrieval of semantically similar patterns rather than requiring exact positional matches within a sliding window.

### Can I use Engram layers with any Transformer architecture?

Engram layers are specific to the Needle 2 implementation in cactus-compute/needle. However, the concept requires only three adaptations for other architectures: implementing the `Engram` module with its embedding tables and convolutional taps, modifying the block forward pass to accept `engram_kv` parameters, and inserting the gating logic shown in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines 319-325.

### How do I tune the engram_conv_taps and engram_conv_dilation parameters?

The `engram_conv_taps` parameter controls the receptive field of the convolution applied to retrieved values, while `engram_conv_dilation` determines the spacing between taps. Start with the defaults (4 taps, dilation 2) and increase taps if your sequences require smoother temporal blending of retrieved context, or adjust dilation to capture patterns at different time scales without increasing the parameter count of the convolutional filter.