Configurable Parameters for the Engram Layer in Needle: Complete Technical Reference

TLDR: The Engram layer in Needle exposes eight primary configurable parameters—including engram_orders, engram_slots, and engram_layers—defined in the TransformerConfig dataclass in needle/model/architecture.py, which control n-gram hashing, per-order head allocation, memory slot capacity, and dilated convolution over value projections.

The Engram layer is a specialized memory‑augmented component in the Needle transformer architecture (cactus-compute/needle) that enhances context processing through hashed n‑gram tables. Tuning these configurable parameters for the Engram layer allows developers to balance memory consumption, computational throughput, and model capacity. This guide examines each parameter's location in the source code, default values, and runtime implications.

Primary Configuration Parameters in TransformerConfig

The user‑facing configuration for the Engram layer resides in the TransformerConfig dataclass inside needle/model/architecture.py (lines 70–76). These five fields determine the high‑level geometry of the memory tables.

N‑gram Orders and Head Allocation

engram_orders defines the tuple of n‑gram sizes used to hash token sequences. According to the source definition at line 72, the default value is (2, 3), meaning the model maintains separate hash tables for bigrams and trigrams.

engram_heads controls the number of attention heads allocated per n‑gram order. Defined at line 73 with a default of 0, the value triggers automatic computation when left unset. The implementation calculates heads as max(1, d_model // (len(engram_orders) * ENGRAM_SUB_DIM)), ensuring the sub‑dimension per head remains consistent across model sizes.

Memory Capacity and Slot Allocation

engram_slots sets the size of each hash table (number of slots). The default value of 8192 at line 74 directly impacts the model's ability to store unique n‑gram contexts. Larger values increase capacity but linearly increase memory consumption.

engram_layers specifies which transformer layers receive Engram memory injections. Defined at line 75 as a tuple of integers defaulting to (2, 15), this parameter determines the depth at which the augmented memory mechanism operates.

Convolution Configuration

engram_conv_dilation controls the dilation factor for the convolution applied to the value stream. While engram_conv_taps is fixed as a constant (see below), the dilation parameter at line 187 allows adjustment of the receptive field over cached values.

Derived Constants and Internal Geometry

Several critical values are computed from the primary parameters above rather than being set directly.

Sub‑Dimension and Table Count

The sub_dim (sub‑dimension) represents the width of each slice within an Engram table. The source code at lines 120–124 derives this as d_model // (len(engram_orders) * engram_heads). This ensures that concatenating all table outputs restores the original model dimensionality.

The num_tables value, referenced at line 490, equals len(engram_orders) * engram_heads. This derived property represents the total count of parallel hash tables maintained by the layer.

Fixed Architecture Constants

engram_conv_taps is a fixed architecture constant defined in needle/model/export.py at line 22 with a value of 4. This setting controls the number of convolution taps applied to the value projection and is baked into the exported model binary format.

dtype specifies the numerical precision for Engram parameters. Implemented at line 188 of architecture.py, the layer defaults to jnp.bfloat16 to optimize memory bandwidth during attention computations.

Implementing Custom Engram Configurations

Developers can override defaults through the TransformerConfig dataclass or, for low‑level experimentation, instantiate the Engram module directly.

Configuring via TransformerConfig

The standard entry point for adjusting Engram layer parameters is the model configuration object:

from needle.model.architecture import TransformerConfig

# Customize Engram geometry for a 1B parameter model

cfg = TransformerConfig(
    d_model=1024,
    engram_orders=(2, 4, 8),      # Extend to 4-grams and 8-grams

    engram_heads=4,               # Override auto-computation

    engram_slots=16384,           # Double the default capacity

    engram_layers=(1, 5, 9),      # Inject at layers 1, 5, and 9

    engram_conv_dilation=2,       # Apply dilated convolution

)

# Instantiate model (pseudo-code; actual class varies by training script)

model = NeedleModel(cfg)

Source: TransformerConfig definition and default values—[architecture.py](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L70-L76).

Direct Engram Module Instantiation

For unit testing or custom architectures, instantiate the Engram class directly using the derived geometry values:

import jax.numpy as jnp
from needle.model.architecture import Engram

# Manually specify computed geometry

engram = Engram(
    d_model=512,
    num_tables=8,          # e.g., 2 orders × 4 heads

    slots=8192,
    sub_dim=16,            # Must equal d_model // (orders × heads)

    num_layers=27,         # Total transformer layers

    conv_dilation=1,
    dtype=jnp.bfloat16,
)

# Dummy forward inputs

indices = jnp.zeros((1, 10, 8), dtype=jnp.int32)   # (batch, seq, num_tables)

ngram_ok = jnp.ones_like(indices[..., :1])          # Validity mask

tap_ok = jnp.ones((4, 1))                          # ENGRAM_CONV_TAPS=4

k, v = engram(indices, ngram_ok, tap_ok, quant=False)

Source: Engram class signature—[architecture.py](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L81-L88).

Summary

• Primary parameters (engram_orders, engram_heads, engram_slots, engram_layers, engram_conv_dilation) are defined in needle/model/architecture.py lines 72–75 and 187, controlling hashing strategy, memory capacity, and injection points.
• Derived values (sub_dim, num_tables) are computed from primary configs at lines 120–124 and 490, ensuring dimensional consistency across the model width.
• Fixed constants (engram_conv_taps=4, dtype=jnp.bfloat16) are hardcoded at export.py line 22 and architecture.py line 188.
• Override defaults via TransformerConfig for standard training workflows, or instantiate the Engram class directly for experimental architectures.

Frequently Asked Questions

What is the default memory capacity for each Engram hash table?

The default engram_slots value is 8192, defined in needle/model/architecture.py at line 74. This sets the number of slots per hash table, with total memory consumption scaling by num_tables × slots × sub_dim.

How does Needle automatically calculate the number of Engram heads?

When engram_heads is set to 0 (the default), the initialization code at line 73 computes heads as max(1, d_model // (len(engram_orders) * ENGRAM_SUB_DIM)). This formula divides the model dimension by the product of order count and a fixed sub‑dimension constant to determine parallel attention heads.

Where is the convolution tap constant defined for Engram processing?

The engram_conv_taps constant is hardcoded to 4 in needle/model/export.py at line 22. This value is serialized into the exported model header and determines the number of convolutional taps applied to value projections during inference.

Can I use n‑gram orders other than the default (2, 3) in the Engram layer?

Yes. The engram_orders parameter accepts any tuple of integers. For example, configuring (2, 4, 8) enables bigram, 4‑gram, and 8‑gram hashing simultaneously, though this increases the total table count and memory requirements proportionally.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}