# How the Engram Module Hashes Token N-grams in Needle

> Discover how the Engram module hashes token n-grams in Needle using a learned hash table and linear projection. Understand the mapping to fixed-size slot indices.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-30

---

**The Engram module implements a learned hash table that maps token n-grams to fixed-size slot indices using a linear projection followed by a deterministic non-linear rounding function.**

The Engram module, located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) within the cactus-compute/needle repository, provides a differentiable mechanism for hashing variable-length token sequences into a fixed address space. Unlike traditional hash tables that require infinite scaling with vocabulary size, this learned approach compresses n-gram representations into a constant number of trainable memory slots through backpropagation-optimized addressing.

## Learned Hashing Architecture in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)

The hashing implementation centers on the `Engram` class (lines 181–210), which transforms sequences of token IDs into integer indices suitable for memory retrieval. The architecture processes discrete tokens through continuous embeddings before applying a trainable discretization step.

### Token Embedding and N-gram Packing

Each incoming token ID is first converted to a dense vector via `self.token_emb`, a shared embedding matrix accessible to the module. For an n-gram of order *n*, the individual token embeddings are aggregated—either concatenated or summed—to form a single vector representation that encapsulates the semantic content of the entire n-gram sequence.

### Projection to Engram Slots

The aggregated n-gram vector passes through a linear projection layer (`self.proj`) that maps from the embedding dimension to the number of available memory slots specified by `cfg.engram_slots`. This projection outputs a dense score for every potential slot, creating a continuous distribution over the fixed-size address space.

### The Hashing Operation

The **hashing** operation applies a deterministic non-linear discretization to the projected scores. According to the implementation at lines 181–210, the module typically applies a `torch.nn.Sigmoid` activation followed by `torch.round`, which maps continuous values to integer indices in the range `[0, cfg.engram_slots-1]`. This rounding operation yields the final slot address used to retrieve learned "slot embeddings" from the Engram memory table.

## Implementing the Hashing Flow

The `hash_ngrams` method exposes the complete transformation from token IDs to discrete addresses. The following example demonstrates module initialization and direct hashing:

```python
import torch
from needle.model.architecture import Engram, Config

# Configuration matching the Engram requirements (simplified)

cfg = Config(
    d_model=512,
    engram_slots=1024,
    ngram_order=3,          # trigrams

    heads=8,
    # additional fields omitted for clarity

)

# Initialize the Engram module

engram = Engram(
    cfg.d_model, 
    cfg.ngram_order * cfg.heads,
    cfg.engram_slots, 
    cfg
)

# Example trigram token IDs (e.g., IDs 12, 45, 78)

token_ids = torch.tensor([[12, 45, 78]])      # shape: (batch, ngram)

# Compute slot indices through learned hashing

slot_indices = engram.hash_ngrams(token_ids)

print("Slot indices:", slot_indices)   # tensor of shape (batch, 1)

```

## Integrating Engram Hashing in Model Forward Passes

During inference, the module processes entire sequences through the `unfold_ngrams` method, which extracts sliding windows before hashing. Lines 492–511 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py) demonstrate this integration within the broader model architecture:

```python
def forward(x):
    # x: token IDs (batch, seq_len)

    # Extract sliding n-grams from the sequence

    ngrams = engram.unfold_ngrams(x)          # (batch, seq_len-n+1, n)

    
    # Hash to discrete slot indices

    slots = engram.hash_ngrams(ngrams)        # (batch, seq_len-n+1, 1)

    
    # Retrieve learned embeddings from the Engram table

    slot_embeddings = engram.slot_embeddings[slots.squeeze(-1)]
    # slot_embeddings now contain the learned representation for each n-gram

    return slot_embeddings

```

The `unfold_ngrams` operation creates overlapping n-gram windows, which `hash_ngrams` maps to discrete addresses. This architecture allows constant-time retrieval of n-gram representations at every position without scaling memory requirements with sequence length.

## Summary

- The `Engram` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 181–210) implements learned hashing for token n-grams through a trainable projection and rounding pipeline.
- **Token n-grams** are embedded, aggregated, and projected into a space with dimensionality equal to `cfg.engram_slots`.
- The **hashing mechanism** uses `Sigmoid` activation followed by `torch.round` to map continuous projections to discrete indices in `[0, cfg.engram_slots-1]`.
- The `hash_ngrams` method handles individual n-grams, while `unfold_ngrams` processes sliding windows across full sequences.
- This design enables constant-memory n-gram retrieval while allowing the model to optimize hash collisions through standard backpropagation.

## Frequently Asked Questions

### What is the Engram module in the cactus-compute/needle repository?

The Engram module is a neural memory component defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) that stores learned representations of token n-grams in a fixed-size hash table. It replaces external memory banks that scale with vocabulary size, instead using learned addressing to compress arbitrary n-grams into a constant number of trainable slots.

### How does the learned hashing differ from traditional hash functions?

Traditional hash functions like MD5 or MurmurHash produce deterministic but non-differentiable outputs based on bitwise operations. The Engram's learned hashing uses a differentiable linear projection (`self.proj`) followed by rounding, allowing gradient flow during training. This enables the model to adaptively organize semantically similar n-grams into identical or adjacent slots through backpropagation.

### What determines the number of available slots in the Engram table?

The slot count is controlled by the `engram_slots` configuration parameter (`cfg.engram_slots`), passed during initialization as shown in lines 492–511 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py). This parameter defines the projection output dimension and the size of the slot embedding matrix, directly constraining the memory capacity for n-gram storage regardless of input vocabulary size.

### Where is the hashing logic implemented in the Needle source code?

The core hashing logic resides in the `Engram` class at lines 181–210 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), specifically within the `hash_ngrams` method. The module construction and its integration into the broader model architecture occur at lines 492–511, where the Engram is instantiated with configuration parameters governing the hash space size.