How Needle 2 Computes Engram Indices: A Deep Dive into the Hashing Algorithm

Needle 2 computes engram indices by applying a deterministic 32-bit hash function that mixes token IDs with order-dependent bit-shifts and a prime multiplier, then reduces the result modulo the table size to generate per-head slot addresses.

Engram indices are the addressing mechanism behind Needle 2's memory-efficient attention system. Unlike standard transformer architectures that compute attention solely from hidden states, Needle stores contextual information in engrams—learned tables indexed by hashed n-gram contexts. The computation of these indices is handled entirely within the engram_indices function in needle/model/architecture.py, which transforms input token IDs into integer slot addresses through a series of bit-wise operations designed for reproducibility and collision resistance.

The engram_indices Function Architecture

The core implementation resides in needle/model/architecture.py and accepts four parameters that define the hashing geometry:

• tokens: Input token IDs (integer array of shape (batch, seq))
• orders: Tuple of n-gram orders (e.g., (2, 3) for bi-grams and tri-grams)
• heads: Number of attention heads (integer)
• slots: Size of each engram embedding table (integer)

The function returns a tensor of shape (batch, seq, orders × heads) containing the computed slot indices for every position, order, and attention head combination.

Step-by-Step Index Computation

The hashing algorithm proceeds through nine distinct stages, each designed to maximize entropy while maintaining deterministic behavior across hardware platforms.

1. Type Conversion to Unsigned 32-bit Integers

The process begins by casting input tokens to uint32 to ensure consistent bit-wise behavior across different hardware architectures and JAX backends.

u = tokens.astype(jnp.uint32)

This guarantees that subsequent shift and XOR operations produce identical results on CPU, GPU, and TPU.

2. Per-Order and Per-Head Iteration

The algorithm iterates over each engram order (representing different n-gram window sizes) and each attention head to generate independent index streams. This nesting ensures that each head receives a unique hashing trajectory, significantly reducing collision probability across the parallel attention mechanisms.

for oi, order in enumerate(orders):
    for h in range(heads):
        # Per-head seed derivation

        seed = (_ENGRAM_SEED * (oi * heads + h + 1)) & 0xFFFFFFFF
        acc = jnp.full_like(u, jnp.uint32(seed))

The seed incorporates the global constant _ENGRAM_SEED combined with the linearized order-head index, creating a unique starting point for each hash stream while maintaining reproducibility.

3. N-gram Hash Mixing

For each position within the current n-gram window, the accumulator undergoes a cryptographic-style mixing operation. The algorithm XORs the accumulator with the token ID shifted right by j positions, then multiplies by the prime constant _ENGRAM_PRIME.

for j in range(order):
    acc = (acc ^ _shift_right(u, j)) * jnp.uint32(_ENGRAM_PRIME)

This step distributes token information across the 32-bit space, where _shift_right creates n-gram sensitivity by incorporating contextual positions through bit displacement rather than explicit window slicing.

4. Final Diffusion and Slot Reduction

After the mixing loop, the accumulator undergoes a final decorrelation step before modular reduction:

acc = acc ^ (acc >> jnp.uint32(15))
idx.append((acc % jnp.uint32(slots)).astype(jnp.int32))

The XOR-shift operation (acc ^ (acc >> 15)) provides additional avalanche effect, ensuring that small changes in input tokens propagate to multiple output bits. The modulo operation maps the 32-bit hash into the concrete table address space [0, slots-1].

5. Tensor Stacking

Finally, all per-head indices are concatenated along the last axis:

return jnp.stack(idx, axis=1)

This produces the output tensor of shape (batch, seq, orders × heads), where the last dimension contains the slot indices for every engram configuration at every sequence position.

Practical Usage Examples

Direct Index Computation

You can invoke the engram_indices function directly to inspect how specific token sequences map to table slots:

import jax.numpy as jnp
from needle.model.architecture import engram_indices

# Sample token IDs (batch=1, sequence length=4)

tokens = jnp.array([[101, 2023, 2003, 102]])

# Engram configuration matching default model setup

orders = (2, 3)      # Bi-gram and tri-gram contexts

heads = 8            # Eight attention heads

slots = 8192         # Table size per head

indices = engram_indices(tokens, orders, heads, slots)
print(f"Shape: {indices.shape}")  # (1, 4, 16)

print(f"Slot range: [{indices.min()}, {indices.max()}]")  # Within [0, 8191]

Integration with the Engram Module

Within the model forward pass, these indices feed directly into the Engram class instances. The engram_geometry helper extracts configuration parameters from the model config:

from needle.model.architecture import Engram, engram_geometry

# Extract parameters from loaded configuration

orders, heads, slots = engram_geometry(model_config)

# Compute indices during forward pass

indices = engram_indices(input_tokens, orders, heads, model_config.engram_slots)

# Retrieve key-value pairs from each engram table

kv_outputs = [
    engram_layer(indices, ngram_mask=valid_ngrams, tap_mask=valid_taps)
    for engram_layer in self.engram_layers
]

Integration with the Needle 2 Architecture

The computed indices serve as the addressing mechanism for several critical subsystems beyond the forward pass. In needle/model/decode.py, the indices are incorporated into the KV cache during autoregressive generation, allowing the model to retrieve previously computed engram contexts without recomputing hashes for the entire prefix.

Configuration management occurs in needle/model/export.py, which extracts engram geometry (orders, heads, slots) from saved model checkpoints, ensuring that exported models preserve the exact hashing parameters used during training.

Summary

• Engram indices are deterministic 32-bit hashes computed in needle/model/architecture.py via the engram_indices function.
• The algorithm uses type-safe uint32 operations, per-head seeding, and prime-based mixing to generate unique slot addresses for each attention head and n-gram order.
• Indices are produced by XORing token IDs with shifted versions of themselves, multiplying by _ENGRAM_PRIME, applying final bit diffusion, and reducing modulo the table size.
• The resulting tensor of shape (batch, seq, orders × heads) feeds directly into the Engram module to retrieve context-aware key-value embeddings for the attention mechanism.

Frequently Asked Questions

What are engram indices used for in Needle 2?

Engram indices function as addresses into learned embedding tables that store contextual information. According to the source code in needle/model/architecture.py, these indices allow the model to retrieve n-gram specific key and value vectors for each attention head, effectively compressing context into fixed-size tables rather than computing full attention over the entire sequence history.

Why does the algorithm use uint32 and bit-wise operations instead of standard hashing?

The implementation casts tokens to uint32 and employs XOR shifts and prime multiplication to guarantee deterministic, hardware-independent behavior. Unlike Python's built-in hash() function, which may vary between sessions or platforms, the 32-bit integer operations in engram_indices produce identical slot addresses across CPU, GPU, and TPU devices, ensuring reproducible model behavior.

How does per-head seeding prevent index collisions?

By deriving a unique seed for each combination of engram order and attention head using the formula (_ENGRAM_SEED * (oi * heads + h + 1)), the algorithm ensures that different heads hash the same token sequence into different areas of the slot space. This separation prevents multiple heads from competing for the same table entries, effectively partitioning the engram memory banks by attention head.

Where is the engram geometry (orders, heads, slots) defined in the codebase?

The geometry parameters are extracted from model configurations in needle/model/export.py (lines 120-131) and utilized during initialization. The engram_geometry helper function parses these values from the model config, which are then passed to engram_indices during both training in architecture.py and inference in decode.py (lines 156-165).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}