# How Multi-Lane Hyper-Connections Work in Needle 2: Architecture and Implementation

> Discover how multi-lane hyper-connections in Needle 2 enhance cross-layer information flow through learned linear transformations, boosting model performance without added depth.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-14

---

**Multi-lane hyper-connections in Needle 2 split each hidden representation into parallel "lanes" that are mixed via learned linear transformations before and after attention blocks, enabling richer cross-layer information flow without increasing model depth.**

Needle 2 introduces multi-lane hyper-connections (MHC) to enhance transformer layer communication by routing information through multiple parallel streams. According to the cactus-compute/needle source code, this mechanism is implemented through lane-specific linear maps defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), allowing each layer to modulate information flow using distinct learned parameters per lane.

## Configuration and Lane Initialization

### Defining the Lane Count in TransformerConfig

The **hyper-parameter** `mhc_lanes` in `TransformerConfig` controls how many parallel lanes are created. By default, this value is set to 4.

```python

# needle/model/architecture.py

mhc_lanes: int = 4

```

This configuration value determines the dimensionality of the lane matrix and the number of distinct parameter sets learned by the model.

### Building the Lane Matrix in Stack.__call__

Inside the `Stack.__call__` method, the model constructs a one-hot "lane" matrix that assigns each layer to a specific lane using modulo arithmetic. This matrix is stored alongside bias offsets that provide each lane with distinct modulation terms.

```python
n, L, nC = cfg.mhc_lanes, cfg.num_layers, cfg.mhc_lanes * cfg.d_model
lane = np.eye(n, dtype=np.float32)[np.arange(L) % n]

hc = {
    "phi_pre":  ...,
    "phi_post": ...,
    "phi_res":  ...,
    "pre_off":  jnp.asarray(8 * lane - 4),
    "post_off": jnp.asarray(-4 * (1 - lane)),
    ...
}

```

The `lane` tensor uses `np.arange(L) % n` to cycle through lane indices (0 to `mhc_lanes-1`) across the layer stack. The `pre_off` and `post_off` terms generate lane-specific bias values—ranging from -4 to 4 for pre-offsets and 0 to -4 for post-offsets—that modulate the hyper-connection gates.

## Hyper-Connection Gate Computation

The actual mixing occurs within `_ScanBody.__call__`, which computes three distinct transformation matrices for each token position.

### Pre-Mix and Post-Mix Gating

The **pre-mix** gate (`hpre`) and **post-mix** gate (`hpost`) apply sigmoid activations to linear projections of the input, incorporating the lane-specific offsets.

```python
hpre = nn.sigmoid(
    hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))
    + hc["b_pre"] + hc["pre_off"]
)

hpost = 2 * nn.sigmoid(
    hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))
    + hc["b_post"] + hc["post_off"]
)

```

The `phi_pre` and `phi_post` matrices are learned parameters with shape `(L, nC, n)`, where `n` is the number of lanes and `nC` is the total dimension (`mhc_lanes * d_model`). The post-mix gate is scaled by 2 to allow the output range to extend beyond [0, 1].

### Residual Mixing with Sinkhorn Normalization

The **residual-mix** gate (`hres`) employs a doubly-stochastic matrix computed via Sinkhorn iterations, enabling structured mixing across lanes.

```python
res = nx @ hc["phi_res"].astype(jnp.float32)
hres = _sinkhorn(hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])

```

The `_sinkhorn` function performs iterative normalization to ensure the mixing matrix remains doubly stochastic. This creates a soft permutation of information across the `mhc_lanes` parallel streams.

### Combining Gates to Update Hidden States

The final hidden state update combines all three gates using einsum operations for efficient tensor contraction.

```python
new_x = (
    jnp.einsum("btij,btjc->btic", hres, xf)
    + hpost[..., None] * y.astype(jnp.float32)[:, :, None, :]
).astype(self.dtype)

```

Here, `hres` mixes the transformed inputs `xf` across lanes, while `hpost` scales the attention block output `y` before broadcasting across lanes. This results in a new hidden state where information has flowed through multiple parallel pathways.

## Enabling and Inspecting Multi-Lane Hyper-Connections

To use MHC in your model, instantiate `TransformerConfig` with your desired lane count and create a `Stack`.

```python
from needle.model.architecture import TransformerConfig, Stack
import jax.numpy as jnp

cfg = TransformerConfig(
    mhc_lanes=6,
    num_layers=12,
    d_model=512,
    num_heads=8,
    num_kv_heads=4
)
stack = Stack(cfg)

# Dummy input (batch=1, seq_len=4, d_model)

x = jnp.zeros((1, 4, cfg.d_model))
out, _ = stack(x)  # Runs with 6-lane hyper-connections

```

After initialization, you can access the learned hyper-connection parameters through the parameter tree.

```python
params = stack.init(jax.random.PRNGKey(0), x)[1]
phi_pre = params['Stack_0']['layers']['mhc_phi_pre']  # shape (L, nC, n)

```

## Summary

- **Multi-lane hyper-connections** split transformer layers into parallel streams controlled by `TransformerConfig.mhc_lanes` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- The `Stack` class initializes a one-hot lane matrix and lane-specific bias offsets (`pre_off`, `post_off`) to differentiate each stream.
- Three gating mechanisms—**pre-mix**, **post-mix**, and **residual-mix**—are computed in `_ScanBody.__call__` using learned `phi_*` matrices and Sinkhorn normalization.
- The final hidden state combines these gates via Einstein summation, enabling rich cross-layer communication without additional depth.
- Configuration requires only setting the `mhc_lanes` hyper-parameter, with parameters accessible through the standard JAX parameter tree.

## Frequently Asked Questions

### What is the default number of lanes in Needle 2 multi-lane hyper-connections?

The default value for `mhc_lanes` is **4**, as defined in the `TransformerConfig` dataclass in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). You can increase this value to create more parallel streams, though this increases the parameter count proportionally.

### How does the lane assignment work across transformer layers?

Layers are assigned to lanes cyclically using modulo arithmetic: layer index modulo `mhc_lanes`. This creates a repeating pattern (e.g., for 6 lanes: 0,1,2,3,4,5,0,1...) that ensures an even distribution of lane-specific transformations across the model depth.

### Why does the residual mixing use Sinkhorn normalization instead of softmax?

The `_sinkhorn` function enforces **doubly-stochastic** constraints on the mixing matrix, meaning both rows and columns sum to 1. This creates a soft permutation matrix that preserves information magnitude during cross-lane mixing more effectively than standard softmax, which only enforces row-wise normalization.

### Can I disable multi-lane hyper-connections while keeping the rest of the architecture?

While the source code defaults to 4 lanes, setting `mhc_lanes=1` effectively disables the multi-lane aspect by collapsing the lane dimension. However, the hyper-connection mechanism (pre, post, and residual gates) still operates as a single-stream transformation, so full disablement would require architectural modifications to skip the gating logic entirely.