# Needle Multi-Lane Hyper-Connections: Architecture and Implementation Guide

> Discover Needle multi-lane hyper-connections. This guide explains how parallel lanes in transformer layers enhance information flow and model performance. Explore the architecture and implementation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-15

---

**Needle multi-lane hyper-connections split each transformer layer's hidden representation into parallel "lanes" that learn lane-specific linear transformations, enabling richer cross-layer information flow without increasing model depth.**

The cactus-compute/needle repository implements multi-lane hyper-connections (MHC) to enhance intra-layer communication through configurable parallel streams. This mechanism, defined primarily in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), uses a lane-specific routing system that modulates attention block inputs and outputs through learned gating parameters.

## What Are Needle Multi-Lane Hyper-Connections?

Needle multi-lane hyper-connections partition each hidden state into **mhc_lanes** parallel streams, where every layer belongs to a specific lane determined by its depth index. Instead of a single transformation path, the model learns distinct **phi** matrices for each lane that control how information mixes before and after the main attention computation.

### Configuration Parameters

The architecture exposes the number of lanes through `TransformerConfig.mhc_lanes`, which defaults to 4:

```python

# needle/model/architecture.py (line 76)

mhc_lanes: int = 4

```

Increasing this value creates more parallel pathways for information routing, allowing the model to specialize different layers into distinct functional groups without adding parameters to the attention mechanisms themselves.

## Lane Preparation in `Stack.__call__`

Within the `Stack` class initialization, Needle constructs a one-hot lane matrix that assigns each layer to a specific lane based on its index modulo `mhc_lanes`. This occurs in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 93-95):

```python
n, L, nC = cfg.mhc_lanes, cfg.num_layers, cfg.mhc_lanes * cfg.d_model
lane = np.eye(n, dtype=np.float32)[np.arange(L) % n]

```

The `lane` tensor indicates which lane a given layer belongs to (computed as `layer % mhc_lanes`). The code then initializes lane-specific offsets:

```python
hc = {
    "phi_pre":  ...,
    "phi_post": ...,
    "phi_res":  ...,
    "pre_off":  jnp.asarray(8 * lane - 4),
    "post_off": jnp.asarray(-4 * (1 - lane)),
    ...
}

```

These `pre_off` and `post_off` values provide distinct bias terms for each lane, later modulating the hyper-connection gates to create differentiable routing behaviors across the parallel streams.

## Hyper-Connection Computation in `_ScanBody.__call__`

The actual mixing occurs inside `_ScanBody.__call__` (lines 51-73), which applies three distinct gating mechanisms to transform the hidden states.

### Pre-Mix Gating (`hpre`)

Before the attention block, the model computes a sigmoid-activated gate using lane-specific parameters:

```python
hpre = nn.sigmoid(
    hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))
    + hc["b_pre"] + hc["pre_off"]
)

```

This gate modulates the input to the attention block based on the lane-specific projection `phi_pre` and the pre-computed offset.

### Post-Mix Gating (`hpost`)

After the attention block, a second gate scales the block's output:

```python
hpost = 2 * nn.sigmoid(
    hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))
    + hc["b_post"] + hc["post_off"]
)

```

The scaling factor of 2 allows the gate to approach 2.0 for strong residual connections or near 0.0 for lane-specific suppression.

### Residual-Mix with Sinkhorn Normalization (`hres`)

The residual pathway uses a specialized mixing matrix with Sinkhorn normalization to maintain valid routing weights:

```python
res   = nx @ hc["phi_res"].astype(jnp.float32)
hres  = _sinkhorn(hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])

```

The `_sinkhorn` function performs iterative normalization to ensure the mixing matrix remains doubly stochastic, balancing information flow across all lanes.

### Final Combination

The three gates combine to produce the new hidden state (lines 71-72):

```python
new_x = (jnp.einsum("btij,btjc->btic", hres, xf)
         + hpost[..., None] * y.astype(jnp.float32)[:, :, None, :]).astype(self.dtype)

```

Here, `hres` mixes the lane-partitioned features `xf` through the einsum operation, while `hpost` scales the attention output `y` before addition.

## Practical Implementation Examples

### Configuring Multiple Lanes

To enable 6 lanes in a 12-layer model:

```python
from needle.model.architecture import TransformerConfig, Stack
import jax.numpy as jnp

cfg = TransformerConfig(mhc_lanes=6, num_layers=12, d_model=512,
                        num_heads=8, num_kv_heads=4)
stack = Stack(cfg)

x = jnp.zeros((1, 4, cfg.d_model))
out, _ = stack(x)  # Runs with 6-lane hyper-connections

```

### Inspecting Learned Parameters

After initialization or training, access the lane-specific projection matrices:

```python
params = stack.init(jax.random.PRNGKey(0), x)[1]
phi_pre = params['Stack_0']['layers']['mhc_phi_pre']  # shape (L, nC, n)

```

The `phi_pre` tensor contains the learned pre-mix projections for each layer, with dimensions corresponding to (number of layers, model dimension × lanes, number of lanes).

## Summary

- **Needle multi-lane hyper-connections** create parallel information pathways by splitting hidden representations into configurable lanes via `TransformerConfig.mhc_lanes`.
- Lane assignment uses modular arithmetic (`layer % mhc_lanes`) in `Stack.__call__` to generate one-hot lane matrices and offset vectors.
- Three distinct gates—**pre-mix**, **post-mix**, and **residual-mix**—control information flow before, during, and after the attention block in `_ScanBody.__call__`.
- The **Sinkhorn normalization** in the residual gate ensures stable routing weights across lanes.
- All hyper-connection parameters are learnable and accessible through the model's parameter dictionary under keys like `mhc_phi_pre`.

## Frequently Asked Questions

### What is the default value for `mhc_lanes` in Needle?

The default value is **4**, defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at line 76 within the `TransformerConfig` dataclass. You can increase this value to create more parallel streams, though this increases the memory footprint for the hyper-connection parameters.

### How does the lane matrix determine which layer belongs to which lane?

The lane matrix uses modular indexing: `np.arange(L) % n` where `L` is the total number of layers and `n` is `mhc_lanes`. This assigns layer 0 to lane 0, layer 1 to lane 1, up to lane `n-1`, then cycles back to lane 0 for layer `n`. The resulting one-hot encoding creates distinct `pre_off` and `post_off` bias terms for each lane.

### What is the purpose of the Sinkhorn normalization in the residual-mix gate?

The `_sinkhorn` function ensures the residual mixing matrix `hres` remains **doubly stochastic**, meaning each row and column sums to 1.0. This normalization prevents any single lane from dominating the residual pathway and maintains balanced information exchange across all parallel streams during the mixing operation.

### How can I access the learned hyper-connection parameters after training?

Access the parameters through the Flax parameter tree using keys like `'mhc_phi_pre'`, `'mhc_phi_post'`, and `'mhc_phi_res'`. After calling `stack.init()` or loading a checkpoint, these tensors reside in the `params` dictionary under the `Stack` layer namespace, with shapes reflecting the lane configuration (typically `(num_layers, d_model * mhc_lanes, mhc_lanes)` for the phi matrices).