# How Multi-Lane Hyper-Connections Work in Needle 2: Architecture and Implementation

> Discover how multi-lane hyper-connections in Needle 2 enhance information flow across transformer layers. Learn about its architecture and implementation in cactus-compute/needle.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-09-12

---

**Multi-lane hyper-connections in Needle 2 split hidden representations into parallel lanes controlled by `mhc_lanes`, applying lane-specific linear transformations through pre-mix, post-mix, and residual-mix gates to enrich information flow across transformer layers without increasing network depth.**

Needle 2, developed in the cactus-compute/needle repository, introduces multi-lane hyper-connections (MHC) to enhance intra-layer communication within transformers. This mechanism divides each layer's hidden state into parallel streams, allowing distinct routing strategies for different information types while maintaining a single forward pass.

## Configuring Multi-Lane Hyper-Connections in TransformerConfig

The architecture is controlled via the `mhc_lanes` parameter in `TransformerConfig`. As defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the default configuration creates four parallel lanes.

```python

# needle/model/architecture.py

mhc_lanes: int = 4               # ↩︎ [config line](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L76)

```

This hyperparameter determines how many parallel "streams" exist within each transformer layer. All subsequent lane-specific transformations derive their dimensions from this value.

## Lane Matrix Initialization in Stack.__call__

The `Stack` class prepares lane assignments during initialization in its `__call__` method. It constructs a one-hot matrix that maps each layer index to its corresponding lane using modulo arithmetic.

```python
n, L, nC = cfg.mhc_lanes, cfg.num_layers, cfg.mhc_lanes * cfg.d_model
lane = np.eye(n, dtype=np.float32)[np.arange(L) % n]   # ↩︎ [Stack init](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L93-L95)

hc = {
    "phi_pre":  …,
    "phi_post": …,
    "phi_res":  …,
    "pre_off":  jnp.asarray(8 * lane - 4),               # ↩︎ [pre_off]

    "post_off": jnp.asarray(-4 * (1 - lane)),           # ↩︎ [post_off]

    …
}

```

The `lane` tensor encodes which lane each layer belongs to via `layer % mhc_lanes`. The `pre_off` and `post_off` arrays inject lane-specific biases into the gating mechanisms, ensuring each parallel stream learns distinct transformation characteristics.

## Computing Hyper-Connection Gates in _ScanBody.__call__

Within the scanned computation body, the `_ScanBody.__call__` method implements three distinct gating operations that mix information across lanes. These gates utilize the lane-specific parameters initialized earlier.

**Pre-mix gate (`hpre`)** applies a sigmoid activation to the input conditioned on lane-specific offsets:

```python
hpre = nn.sigmoid(
    hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))
    + hc["b_pre"] + hc["pre_off"]
)                                         # ↩︎ [pre gate](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L61-L63)

```

**Post-mix gate (`hpost`)** scales the attention block output with a learned transformation:

```python
hpost = 2 * nn.sigmoid(
    hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))
    + hc["b_post"] + hc["post_off"]
)                                         # ↩︎ [post gate](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L66-L68)

```

**Residual-mix gate (`hres`)** uses Sinkhorn normalization to stabilize cross-lane residual connections:

```python
res   = nx @ hc["phi_res"].astype(jnp.float32)
hres  = _sinkhorn(hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])

# ↩︎ [res gate](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L69-L71)

```

The final hidden state combines these gated components via `einsum` operations:

```python
new_x = (jnp.einsum("btij,btjc->btic", hres, xf)
         + hpost[..., None] * y.astype(jnp.float32)[:, :, None, :]).astype(self.dtype)

```

## Practical Implementation Example

To utilize multi-lane hyper-connections, configure `TransformerConfig` and instantiate the `Stack` class as shown below.

```python

# Example: Enable 6 lanes and inspect the lane matrix

from needle.model.architecture import TransformerConfig, Stack
import jax.numpy as jnp

cfg = TransformerConfig(mhc_lanes=6, num_layers=12, d_model=512,
                        num_heads=8, num_kv_heads=4)
stack = Stack(cfg)

# Dummy input (batch=1, seq_len=4, d_model)

x = jnp.zeros((1, 4, cfg.d_model))
out, _ = stack(x)                     # Runs with 6‑lane hyper‑connections

```

After initialization, you can access the learned lane-specific parameters for inspection or analysis:

```python

# Access the learned hyper‑connection parameters after training

params = stack.init(jax.random.PRNGKey(0), x)[1]   # init returns (output, params)

phi_pre = params['Stack_0']['layers']['mhc_phi_pre']   # shape (L, nC, n)

```

The `phi_pre`, `phi_post`, and `phi_res` arrays contain the learned linear maps that enable parallel information routing across the configured lanes.

## Summary

- **Multi-lane hyper-connections** split transformer representations into `mhc_lanes` parallel streams (default 4) to enhance model capacity without increasing depth.
- Lane assignments are computed in `Stack.__call__` via a one-hot matrix using `np.eye(n)[np.arange(L) % n]`, mapping layers to lanes cyclically.
- Three distinct gating mechanisms—**pre-mix**, **post-mix**, and **residual-mix**—modulate information flow using lane-specific offsets and learned transformations.
- The `_sinkhorn` function stabilizes residual connections across lanes, while `einsum` operations efficiently combine the gated outputs.
- Configuration is controlled through `TransformerConfig.mhc_lanes`, making the number of parallel lanes a tunable hyperparameter in the cactus-compute/needle codebase.

## Frequently Asked Questions

### How do multi-lane hyper-connections differ from standard residual connections?

Standard residual connections simply add the layer input to its output. Multi-lane hyper-connections in Needle 2 replace this with a sophisticated routing mechanism where information flows through parallel lanes via learned linear maps (`phi_pre`, `phi_post`, `phi_res`) and three distinct gating operations. This allows the model to maintain separate information pathways that interact selectively rather than following a single monolithic stream.

### What is the purpose of the Sinkhorn normalization in the residual gate?

The `_sinkhorn` function applies iterative normalization to the residual gate matrix `hres` to ensure stable gradients and prevent any single lane from dominating the output. As implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this maintains balanced information flow across all `mhc_lanes` during the residual mixing phase.

### Can the number of lanes exceed the number of layers?

Yes. The lane assignment uses modulo arithmetic (`np.arange(L) % n`), so if `mhc_lanes` is larger than `num_layers`, some lanes simply receive no layer assignments. However, this is generally inefficient as it leaves lane-specific parameters underutilized; the optimal configuration typically sets `mhc_lanes` to a divisor of or smaller than the total layer count.

### Where are the lane-specific offsets initialized in the codebase?

The offsets are computed in `Stack.__call__` within [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 93-95). The `pre_off` tensor is calculated as `8 * lane - 4` and `post_off` as `-4 * (1 - lane)`, where `lane` represents the one-hot matrix of layer-to-lane assignments. These values provide distinct bias terms that differentiate the gating behavior of each parallel stream.