How the Engram Module Hashes Token N-grams in Needle

The Engram module implements a learned hash table that maps token n-grams to fixed-size slot indices using a linear projection followed by a deterministic non-linear rounding function.

The Engram module, located in needle/model/architecture.py within the cactus-compute/needle repository, provides a differentiable mechanism for hashing variable-length token sequences into a fixed address space. Unlike traditional hash tables that require infinite scaling with vocabulary size, this learned approach compresses n-gram representations into a constant number of trainable memory slots through backpropagation-optimized addressing.

Learned Hashing Architecture in needle/model/architecture.py

The hashing implementation centers on the Engram class (lines 181–210), which transforms sequences of token IDs into integer indices suitable for memory retrieval. The architecture processes discrete tokens through continuous embeddings before applying a trainable discretization step.

Token Embedding and N-gram Packing

Each incoming token ID is first converted to a dense vector via self.token_emb, a shared embedding matrix accessible to the module. For an n-gram of order n, the individual token embeddings are aggregated—either concatenated or summed—to form a single vector representation that encapsulates the semantic content of the entire n-gram sequence.

Projection to Engram Slots

The aggregated n-gram vector passes through a linear projection layer (self.proj) that maps from the embedding dimension to the number of available memory slots specified by cfg.engram_slots. This projection outputs a dense score for every potential slot, creating a continuous distribution over the fixed-size address space.

The Hashing Operation

The hashing operation applies a deterministic non-linear discretization to the projected scores. According to the implementation at lines 181–210, the module typically applies a torch.nn.Sigmoid activation followed by torch.round, which maps continuous values to integer indices in the range [0, cfg.engram_slots-1]. This rounding operation yields the final slot address used to retrieve learned "slot embeddings" from the Engram memory table.

Implementing the Hashing Flow

The hash_ngrams method exposes the complete transformation from token IDs to discrete addresses. The following example demonstrates module initialization and direct hashing:

import torch
from needle.model.architecture import Engram, Config

# Configuration matching the Engram requirements (simplified)

cfg = Config(
    d_model=512,
    engram_slots=1024,
    ngram_order=3,          # trigrams

    heads=8,
    # additional fields omitted for clarity

)

# Initialize the Engram module

engram = Engram(
    cfg.d_model, 
    cfg.ngram_order * cfg.heads,
    cfg.engram_slots, 
    cfg
)

# Example trigram token IDs (e.g., IDs 12, 45, 78)

token_ids = torch.tensor([[12, 45, 78]])      # shape: (batch, ngram)

# Compute slot indices through learned hashing

slot_indices = engram.hash_ngrams(token_ids)

print("Slot indices:", slot_indices)   # tensor of shape (batch, 1)

Integrating Engram Hashing in Model Forward Passes

During inference, the module processes entire sequences through the unfold_ngrams method, which extracts sliding windows before hashing. Lines 492–511 of architecture.py demonstrate this integration within the broader model architecture:

def forward(x):
    # x: token IDs (batch, seq_len)

    # Extract sliding n-grams from the sequence

    ngrams = engram.unfold_ngrams(x)          # (batch, seq_len-n+1, n)

    
    # Hash to discrete slot indices

    slots = engram.hash_ngrams(ngrams)        # (batch, seq_len-n+1, 1)

    
    # Retrieve learned embeddings from the Engram table

    slot_embeddings = engram.slot_embeddings[slots.squeeze(-1)]
    # slot_embeddings now contain the learned representation for each n-gram

    return slot_embeddings

The unfold_ngrams operation creates overlapping n-gram windows, which hash_ngrams maps to discrete addresses. This architecture allows constant-time retrieval of n-gram representations at every position without scaling memory requirements with sequence length.

Summary

• The Engram class in needle/model/architecture.py (lines 181–210) implements learned hashing for token n-grams through a trainable projection and rounding pipeline.
• Token n-grams are embedded, aggregated, and projected into a space with dimensionality equal to cfg.engram_slots.
• The hashing mechanism uses Sigmoid activation followed by torch.round to map continuous projections to discrete indices in [0, cfg.engram_slots-1].
• The hash_ngrams method handles individual n-grams, while unfold_ngrams processes sliding windows across full sequences.
• This design enables constant-memory n-gram retrieval while allowing the model to optimize hash collisions through standard backpropagation.

Frequently Asked Questions

What is the Engram module in the cactus-compute/needle repository?

The Engram module is a neural memory component defined in needle/model/architecture.py that stores learned representations of token n-grams in a fixed-size hash table. It replaces external memory banks that scale with vocabulary size, instead using learned addressing to compress arbitrary n-grams into a constant number of trainable slots.

How does the learned hashing differ from traditional hash functions?

Traditional hash functions like MD5 or MurmurHash produce deterministic but non-differentiable outputs based on bitwise operations. The Engram's learned hashing uses a differentiable linear projection (self.proj) followed by rounding, allowing gradient flow during training. This enables the model to adaptively organize semantically similar n-grams into identical or adjacent slots through backpropagation.

What determines the number of available slots in the Engram table?

The slot count is controlled by the engram_slots configuration parameter (cfg.engram_slots), passed during initialization as shown in lines 492–511 of architecture.py. This parameter defines the projection output dimension and the size of the slot embedding matrix, directly constraining the memory capacity for n-gram storage regardless of input vocabulary size.

Where is the hashing logic implemented in the Needle source code?

The core hashing logic resides in the Engram class at lines 181–210 of needle/model/architecture.py, specifically within the hash_ngrams method. The module construction and its integration into the broader model architecture occur at lines 492–511, where the Engram is instantiated with configuration parameters governing the hash space size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}