How Multi-Lane Hyper-Connections Work in Needle 2's Architecture

Needle 2 replaces the traditional feed-forward network with a Simple Attention Network that uses multi-lane hyper-connections to create parallel hidden streams across transformer layers, enabling richer contextual interactions through learned linear maps and gated mixing.

Multi-lane hyper-connections (MHC) represent a fundamental architectural shift in the cactus-compute/needle repository, replacing the conventional residual stream with multiple parallel "lanes" that communicate across layers. This mechanism allows each transformer layer to interact with shared hidden states learned jointly across the stack, providing more expressive downstream representations while maintaining parameter efficiency. Understanding how these hyper-connections configure, route, and gate information is essential for customizing Needle 2 models or interpreting their internal computations.

What Are Multi-Lane Hyper-Connections?

In traditional transformers, information flows through a single residual stream. Multi-lane hyper-connections break this constraint by establishing parallel pathways—configured via the mhc_lanes parameter—through which each layer can broadcast and receive information. According to the source code in needle/model/architecture.py, these connections replace the standard feed-forward network (FFN) with a more flexible mixing mechanism that operates across num_layers positions and mhc_lanes parallel dimensions simultaneously.

Configuration and Lane Construction

Setting the Number of Lanes

The MHC behavior is controlled through the TransformerConfig class, where mhc_lanes defaults to 4 as defined at line 76 of needle/model/architecture.py. This parameter determines how many parallel hidden streams the model will maintain throughout its forward pass.

Cyclic Lane Assignment

Inside the Stack module, the architecture constructs a one-hot matrix named lane that cyclically assigns each of the num_layers positions to one of the available lanes. As implemented at lines 94-95 of needle/model/architecture.py, this assignment ensures that layers are distributed evenly across lanes, creating a repeating pattern that maximizes cross-layer information sharing within each parallel stream.

Learned Hyper-Connection Parameters

For every layer in the stack, Needle 2 initializes a specific set of learnable tensors that govern how information flows between lanes. These parameters—mhc_phi_pre, mhc_phi_post, mhc_phi_res, along with their associated biases and scaling factors—are created with shapes connecting the per-lane hidden dimension n = cfg.mhc_lanes to the full model dimension nC = cfg.num_layers * cfg.mhc_lanes * cfg.d_model (lines 96-104 of needle/model/architecture.py).

The phi matrices serve as the core linear transformations that map information between the global hidden state and individual lane representations, while bias terms provide layer-specific offsets that allow each position in the stack to modulate its lane interactions uniquely.

The Three Types of Hyper-Connections

The _ScanBody.__call__ method at lines 61-73 of needle/model/architecture.py combines three distinct connection mechanisms to form the new representation for each token.

Pre-Connections

Pre-connections utilize phi_pre, a_pre, and b_pre to mix the RMS-normalized input with a lane-specific gate hpre. This operation produces a per-lane signal that is added back to the token stream before the main attention computation, effectively conditioning the input based on the current state of each parallel lane.

Post-Connections

Similarly, post-connections employ phi_post along with a_post and b_post to blend the block output y with a lane-specific gate hpost. This allows the model to re-integrate information from the parallel lanes after the attention mechanism has processed the input, ensuring that computational results are properly distributed across the multi-lane structure.

Residual Connections via Sinkhorn Iteration

The residual hyper-connection uses phi_res and a_res to form a doubly-stochastic matrix through a Sinkhorn iteration implemented in the _sinkhorn function. This matrix re-weights the hidden states before they are summed back into the main stream as new_x, providing a learnable, normalized routing mechanism that maintains stability across deep stacks while allowing flexible information flow between lanes.

Lane-Wise Gating and Broadcasting

Broadcasting Tokens Across Lanes

To enable simultaneous processing across all parallel streams, Needle 2 broadcasts the token tensor from shape (*x.shape[:2], x.shape[-1]) to (*x.shape[:2], n, x.shape[-1]) using jnp.broadcast_to (line 108 of needle/model/architecture.py). This expansion allows the same input tokens to interact with each of the n lanes independently before the hyper-connection mechanisms aggregate the results.

Lane-Specific Bias Terms

The architecture incorporates pre_off and post_off bias terms (lines 105-106 of needle/model/architecture.py) to provide each lane with its own modulation pattern. These offsets ensure that different lanes can specialize in distinct types of information processing, with the bias terms shifting the gating values (hpre and hpost) to create diverse activation patterns across the parallel streams.

Working with Multi-Lane Hyper-Connections in Code

The following examples demonstrate how to inspect and utilize MHC configurations in the Needle 2 architecture.

Inspecting MHC Configuration

from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork
import jax.numpy as jnp

# Define a config with 4 lanes (default) and a small model for demo

cfg = TransformerConfig(
    d_model=256,
    num_heads=4,
    num_kv_heads=2,
    num_layers=6,
    mhc_lanes=4,          # <-- number of hyper-connection lanes

    engram_layers=(2, 4)  # optional engram layers

)

# Build the model

model = SimpleAttentionNetwork(cfg)

# Create a dummy token sequence (batch-size 1, length 8)

tokens = jnp.array([[1, 2, 3, 4, 5, 6, 7, 8]])

# Run a forward pass (logits only)

logits = model(tokens, return_mtp=False)
print("Logits shape:", logits.shape)   # → (1, 8, vocab_size)

Accessing Lane-Specific Gating Parameters

import jax
import numpy as np
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(num_layers=6, mhc_lanes=4, d_model=256)
model = SimpleAttentionNetwork(cfg)

# Access the internal parameters after initialization

params = model.init(jax.random.PRNGKey(0), jnp.ones((1, 1), dtype=jnp.int32))

# The pre-gate bias for layer 0, lane 0:

pre_bias = params["Stack_0"]["layers"]["mhc_b_pre"][0, 0]
print("Pre-gate bias (layer 0, lane 0):", pre_bias)

Summary

• Multi-lane hyper-connections replace the standard FFN in Needle 2's Simple Attention Network, creating mhc_lanes parallel hidden streams (default 4) that are configured in needle/model/architecture.py.
• Lane assignment follows a cyclic pattern via a one-hot matrix constructed in the Stack module (lines 94-95), distributing layers evenly across parallel pathways.
• Three distinct connection types—pre, post, and residual—govern information flow using learned phi matrices and Sinkhorn-normalized routing at lines 61-73.
• Broadcasting operations at line 108 expand token tensors to interact with all lanes simultaneously, while bias terms pre_off and post_off enable lane-specific specialization (lines 105-106).
• The architecture maintains parameter efficiency by sharing hyper-connection logic across layers while allowing each lane to develop distinct representations through gated mixing mechanisms.

Frequently Asked Questions

What is the default number of lanes in Needle 2's MHC architecture?

According to the TransformerConfig definition in needle/model/architecture.py at line 76, the default value for mhc_lanes is 4. This means the model creates four parallel hidden streams by default, though this can be adjusted based on computational requirements and desired model capacity.

How do multi-lane hyper-connections differ from standard transformer residual connections?

Standard transformers use a single residual stream where each layer adds its output to a shared hidden state. In contrast, multi-lane hyper-connections establish multiple parallel pathways (lanes) that interact through learned linear transformations (mhc_phi_pre, mhc_phi_post, mhc_phi_res) and gated mixing. This allows information to route between lanes via the doubly-stochastic residual matrix computed through Sinkhorn iteration, rather than following a strictly sequential additive pattern.

What role does the Sinkhorn iteration play in the residual hyper-connections?

The Sinkhorn iteration, implemented in the _sinkhorn function and utilized in the residual connection logic, normalizes the phi_res parameters to create a doubly-stochastic matrix. This matrix re-weights hidden states before summing them back into the main stream as new_x, ensuring that information from different lanes is combined in a normalized, stable manner that preserves the scale of activations across deep transformer stacks.

Where are the hyper-connection parameters defined in the Needle 2 source code?

All MHC parameters—including mhc_phi_pre, mhc_phi_post, mhc_phi_res, and their associated bias terms—are defined within the Stack module in needle/model/architecture.py at lines 96-104. The actual application of these parameters occurs in the _ScanBody.__call__ method at lines 61-73, while the high-level architecture description referencing "multi-lane hyper-connections" appears in README.md at lines 21-22.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}