# How Multi-Lane Hyper-Connections Work in Needle 2's Architecture

> Explore multi-lane hyper-connections in Needle 2's architecture. Discover how parallel hidden streams and learned linear maps enhance contextual interactions in this advanced neural network design.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-29

---

**Needle 2 replaces the traditional feed-forward network with a Simple Attention Network that uses multi-lane hyper-connections to create parallel hidden streams across transformer layers, enabling richer contextual interactions through learned linear maps and gated mixing.**

Multi-lane hyper-connections (MHC) represent a fundamental architectural shift in the cactus-compute/needle repository, replacing the conventional residual stream with multiple parallel "lanes" that communicate across layers. This mechanism allows each transformer layer to interact with shared hidden states learned jointly across the stack, providing more expressive downstream representations while maintaining parameter efficiency. Understanding how these hyper-connections configure, route, and gate information is essential for customizing Needle 2 models or interpreting their internal computations.

## What Are Multi-Lane Hyper-Connections?

In traditional transformers, information flows through a single residual stream. **Multi-lane hyper-connections** break this constraint by establishing parallel pathways—configured via the `mhc_lanes` parameter—through which each layer can broadcast and receive information. According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), these connections replace the standard feed-forward network (FFN) with a more flexible mixing mechanism that operates across `num_layers` positions and `mhc_lanes` parallel dimensions simultaneously.

## Configuration and Lane Construction

### Setting the Number of Lanes

The MHC behavior is controlled through the `TransformerConfig` class, where `mhc_lanes` defaults to 4 as defined at line 76 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This parameter determines how many parallel hidden streams the model will maintain throughout its forward pass.

### Cyclic Lane Assignment

Inside the `Stack` module, the architecture constructs a one-hot matrix named `lane` that cyclically assigns each of the `num_layers` positions to one of the available lanes. As implemented at lines 94-95 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this assignment ensures that layers are distributed evenly across lanes, creating a repeating pattern that maximizes cross-layer information sharing within each parallel stream.

## Learned Hyper-Connection Parameters

For every layer in the stack, Needle 2 initializes a specific set of learnable tensors that govern how information flows between lanes. These parameters—`mhc_phi_pre`, `mhc_phi_post`, `mhc_phi_res`, along with their associated biases and scaling factors—are created with shapes connecting the per-lane hidden dimension `n = cfg.mhc_lanes` to the full model dimension `nC = cfg.num_layers * cfg.mhc_lanes * cfg.d_model` (lines 96-104 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)).

The **phi matrices** serve as the core linear transformations that map information between the global hidden state and individual lane representations, while bias terms provide layer-specific offsets that allow each position in the stack to modulate its lane interactions uniquely.

## The Three Types of Hyper-Connections

The `_ScanBody.__call__` method at lines 61-73 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) combines three distinct connection mechanisms to form the new representation for each token.

### Pre-Connections

**Pre-connections** utilize `phi_pre`, `a_pre`, and `b_pre` to mix the RMS-normalized input with a lane-specific gate `hpre`. This operation produces a per-lane signal that is added back to the token stream before the main attention computation, effectively conditioning the input based on the current state of each parallel lane.

### Post-Connections

Similarly, **post-connections** employ `phi_post` along with `a_post` and `b_post` to blend the block output `y` with a lane-specific gate `hpost`. This allows the model to re-integrate information from the parallel lanes after the attention mechanism has processed the input, ensuring that computational results are properly distributed across the multi-lane structure.

### Residual Connections via Sinkhorn Iteration

The **residual hyper-connection** uses `phi_res` and `a_res` to form a doubly-stochastic matrix through a Sinkhorn iteration implemented in the `_sinkhorn` function. This matrix re-weights the hidden states before they are summed back into the main stream as `new_x`, providing a learnable, normalized routing mechanism that maintains stability across deep stacks while allowing flexible information flow between lanes.

## Lane-Wise Gating and Broadcasting

### Broadcasting Tokens Across Lanes

To enable simultaneous processing across all parallel streams, Needle 2 broadcasts the token tensor from shape `(*x.shape[:2], x.shape[-1])` to `(*x.shape[:2], n, x.shape[-1])` using `jnp.broadcast_to` (line 108 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)). This expansion allows the same input tokens to interact with each of the `n` lanes independently before the hyper-connection mechanisms aggregate the results.

### Lane-Specific Bias Terms

The architecture incorporates `pre_off` and `post_off` bias terms (lines 105-106 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)) to provide each lane with its own modulation pattern. These offsets ensure that different lanes can specialize in distinct types of information processing, with the bias terms shifting the gating values (`hpre` and `hpost`) to create diverse activation patterns across the parallel streams.

## Working with Multi-Lane Hyper-Connections in Code

The following examples demonstrate how to inspect and utilize MHC configurations in the Needle 2 architecture.

### Inspecting MHC Configuration

```python
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork
import jax.numpy as jnp

# Define a config with 4 lanes (default) and a small model for demo

cfg = TransformerConfig(
    d_model=256,
    num_heads=4,
    num_kv_heads=2,
    num_layers=6,
    mhc_lanes=4,          # <-- number of hyper-connection lanes

    engram_layers=(2, 4)  # optional engram layers

)

# Build the model

model = SimpleAttentionNetwork(cfg)

# Create a dummy token sequence (batch-size 1, length 8)

tokens = jnp.array([[1, 2, 3, 4, 5, 6, 7, 8]])

# Run a forward pass (logits only)

logits = model(tokens, return_mtp=False)
print("Logits shape:", logits.shape)   # → (1, 8, vocab_size)

```

### Accessing Lane-Specific Gating Parameters

```python
import jax
import numpy as np
from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(num_layers=6, mhc_lanes=4, d_model=256)
model = SimpleAttentionNetwork(cfg)

# Access the internal parameters after initialization

params = model.init(jax.random.PRNGKey(0), jnp.ones((1, 1), dtype=jnp.int32))

# The pre-gate bias for layer 0, lane 0:

pre_bias = params["Stack_0"]["layers"]["mhc_b_pre"][0, 0]
print("Pre-gate bias (layer 0, lane 0):", pre_bias)

```

## Summary

- **Multi-lane hyper-connections** replace the standard FFN in Needle 2's Simple Attention Network, creating `mhc_lanes` parallel hidden streams (default 4) that are configured in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- Lane assignment follows a cyclic pattern via a one-hot matrix constructed in the `Stack` module (lines 94-95), distributing layers evenly across parallel pathways.
- Three distinct connection types—pre, post, and residual—govern information flow using learned phi matrices and Sinkhorn-normalized routing at lines 61-73.
- Broadcasting operations at line 108 expand token tensors to interact with all lanes simultaneously, while bias terms `pre_off` and `post_off` enable lane-specific specialization (lines 105-106).
- The architecture maintains parameter efficiency by sharing hyper-connection logic across layers while allowing each lane to develop distinct representations through gated mixing mechanisms.

## Frequently Asked Questions

### What is the default number of lanes in Needle 2's MHC architecture?

According to the `TransformerConfig` definition in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at line 76, the default value for `mhc_lanes` is 4. This means the model creates four parallel hidden streams by default, though this can be adjusted based on computational requirements and desired model capacity.

### How do multi-lane hyper-connections differ from standard transformer residual connections?

Standard transformers use a single residual stream where each layer adds its output to a shared hidden state. In contrast, multi-lane hyper-connections establish multiple parallel pathways (lanes) that interact through learned linear transformations (`mhc_phi_pre`, `mhc_phi_post`, `mhc_phi_res`) and gated mixing. This allows information to route between lanes via the doubly-stochastic residual matrix computed through Sinkhorn iteration, rather than following a strictly sequential additive pattern.

### What role does the Sinkhorn iteration play in the residual hyper-connections?

The Sinkhorn iteration, implemented in the `_sinkhorn` function and utilized in the residual connection logic, normalizes the `phi_res` parameters to create a doubly-stochastic matrix. This matrix re-weights hidden states before summing them back into the main stream as `new_x`, ensuring that information from different lanes is combined in a normalized, stable manner that preserves the scale of activations across deep transformer stacks.

### Where are the hyper-connection parameters defined in the Needle 2 source code?

All MHC parameters—including `mhc_phi_pre`, `mhc_phi_post`, `mhc_phi_res`, and their associated bias terms—are defined within the `Stack` module in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 96-104. The actual application of these parameters occurs in the `_ScanBody.__call__` method at lines 61-73, while the high-level architecture description referencing "multi-lane hyper-connections" appears in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) at lines 21-22.