How Multi-Lane Hyper-Connections and Sinkhorn Iteration Work in Needle 2

Multi-lane hyper-connections split transformer layers into parallel "lanes" with learned gating and Sinkhorn-based residual mixing, while Sinkhorn iteration generates doubly-stochastic routing matrices that redistribute signals across lanes in a mass-preserving, differentiable way.

Needle 2's Simple Attention Network replaces the traditional feed-forward layer with a Hadamard-MLP and introduces these two mechanisms to enable adaptive, efficient routing across parallel processing paths. This article explains exactly how these components function in the source code.

What Are Multi-Lane Hyper-Connections?

Multi-lane hyper-connections (MHC) partition each transformer layer into mhc_lanes independent pathways that process information in parallel. Unlike standard residual connections, these lanes interact through three distinct gating stages, allowing the network to learn dynamic information flow.

The mechanism resides in needle/model/architecture.py, with the core implementation split across two locations:

• Lane broadcasting occurs in Stack.__call__ at lines 93-94
• Per-layer parameters are defined inside _ScanBody in the hc dictionary at lines 68-73

The Five-Stage Pipeline

Stage	Operation	Code Location
Lane broadcasting	Expand x from [B, T, d_model] to [B, T, n, d_model]	Stack.__call__, lines 94-95
Pre-gate (hpre)	Project and gate engram values before injection	_ScanBody, phi_pre, b_pre
Post-gate (hpost)	Scale block outputs before reintegration	_ScanBody, a_post, b_post
Residual mixing (hres)	Sinkhorn-based lane blending	_ScanBody, phi_res → _sinkhorn
Lane aggregation	Sum over lane dimension to restore original shape	Final operation in scan body

Pre-Gate Mechanism

The pre-gate controls how much engram information enters each lane. RMS-normalized embeddings nx pass through a learned linear projection phi_pre, followed by a sigmoid with bias b_pre and offset pre_off:

hpre = sigmoid(nx @ phi_pre + b_pre + pre_off)

This produces per-lane gating weights that scale engram values before they merge with the token stream.

Post-Gate Mechanism

After the main Block computes its update y, the post-gate determines reintegration strength using learned scalars a_post, b_post, and offset post_off. This creates adaptive residual pathways that vary by lane and layer depth.

Residual Mixing with Sinkhorn

The residual term res = nx @ phi_res becomes a routing matrix through the Sinkhorn operator. The resulting hres softly blends lane-wise values (xf) before combination with the gated update, enabling cross-lane information redistribution without hard routing decisions.

How Sinkhorn Iteration Enables Differentiable Routing

The Sinkhorn iteration in Needle 2 solves a critical problem: how to mix lane representations with guaranteed constraints while maintaining differentiability. The private helper _sinkhorn at architecture.py lines 69-74 implements this via log-space normalization.

The Iteration Algorithm

For iters iterations (default 20), the algorithm alternates row and column normalization in log-space:

log_K = log_K - jax.nn.logsumexp(log_K, axis=-1, keepdims=True)  # row normalization

log_K = log_K - jax.nn.logsumexp(log_K, axis=-2, keepdims=True)  # column normalization

Exponentiating the result yields a doubly-stochastic matrix where every row and column sums to exactly 1.

Why Doubly-Stochastic Constraints Matter

This mathematical property ensures three critical behaviors:

1. Mass preservation — No lane arbitrarily amplifies or attenuates total signal magnitude
2. Complete assignment — All residual information is redistributed, none is lost
3. Differentiability — Gradients flow through the entire routing operation for end-to-end learning

The constraint mirrors optimal transport theory, where Sinkhorn iteration provides an efficient approximation to the optimal assignment problem.

Implementation Walkthrough

Activating Multi-Lane Hyper-Connections

Configure mhc_lanes in the transformer config to enable the mechanism:

import jax.numpy as jnp
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

# Build model with 4 hyper-connection lanes

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,
    num_layers=27,
    mhc_lanes=4,          # activates multi-lane hyper-connections

    engram_layers=(2, 15),
)

model = SimpleAttentionNetwork(cfg)

# Forward pass automatically uses MHC

tokens = jnp.arange(0, 16)[None, :]   # shape (1, 16)

logits = model(tokens)                # shape (1, 16, vocab_size)

print("Output shape:", logits.shape)

The mhc_lanes=4 parameter triggers:

• Creation of phi_pre, phi_post, phi_res linear maps per layer
• Initialization of a_post, b_pre, b_post gating scalars
• Integration of the Sinkhorn operator into the forward pass

Direct Sinkhorn Invocation

For debugging or custom routing, access the operator directly:

from needle.model.architecture import _sinkhorn

# Random routing logits: batch=1, 8 lanes mixing to 8 lanes

raw_logits = jnp.ones((1, 8, 8)) * 0.3

# Generate doubly-stochastic mixing matrix

mixing_matrix = _sinkhorn(raw_logits, iters=20)

# Verify constraints

print("Row sums:", mixing_matrix.sum(axis=-1))    # all 1.0

print("Column sums:", mixing_matrix.sum(axis=-2)) # all 1.0

The output matrix can replace or augment standard attention patterns in specialized architectures.

Architecture Files and Their Roles

File	Purpose
needle/model/architecture.py	Core implementation: SimpleAttentionNetwork, _ScanBody, _sinkhorn, and all hyper-connection parameters
needle/model/decode.py	Weight deserialization mapping saved checkpoints to mhc_* tensors
needle/model/export.py	Checkpoint packaging including hyper-connection state
README.md	High-level architecture description with hyper-connection rationale

Summary

• Multi-lane hyper-connections split layers into parallel pathways with three-stage gating (pre-gate, post-gate, residual mixing), implemented in Stack.__call__ and _ScanBody within architecture.py

• Sinkhorn iteration at _sinkhorn (architecture.py lines 69-74) produces differentiable doubly-stochastic matrices via alternating log-space normalization, enabling mass-preserving lane mixing

• The combination replaces rigid residual connections with learned, adaptive routing that allocates computational capacity dynamically across lanes

• All mechanisms are fully differentiable and trained end-to-end with standard gradient descent

Frequently Asked Questions

How many Sinkhorn iterations does Needle 2 use by default?

The default is 20 iterations as specified in the _sinkhorn function signature. This balances convergence accuracy against computational overhead—empirically sufficient for stable doubly-stochastic approximation without excessive latency in the forward pass.

Can I change the number of lanes after training?

No. The mhc_lanes parameter determines the shape of learned tensors (phi_pre, phi_post, phi_res, and all gating scalars) throughout the network. Changing this requires reinitializing these parameters and retraining. The decode.py file maps saved weights to these specific shapes during checkpoint loading.

Why use Sinkhorn instead of softmax for routing?

Softmax normalizes only across rows, permitting arbitrary column sums that can amplify or suppress total signal magnitude. Sinkhorn enforces doubly-stochastic constraints—both dimensions sum to 1—preserving total "mass" across the routing operation. This stabilization proves critical for training deep networks with cross-lane mixing.

Where does the "hyper-connection" name originate?

The term reflects that these connections operate above standard residual pathways, creating higher-order interaction patterns between parallel lanes rather than simple skip connections. The prefix "hyper" indicates the learned, multiplicative gating and structured mixing that distinguishes them from additive residuals.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}