# How Needle 2 Implements Multi-Lane Hyper-Connections: Architecture Deep Dive

> Explore Needle 2's architecture and learn how multi-lane hyper-connections enhance information flow. Discover parallel lanes and gating matrices for richer cross-layer communication.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-31

---

**Needle 2 implements multi-lane hyper-connections by splitting hidden representations into parallel "lanes" and applying lane-specific gating matrices before and after each transformer block's attention computation, enabling richer cross-layer information flow without additional depth.**

Multi-lane hyper-connections (MHC) represent a key architectural innovation in the [cactus-compute/needle](https://github.com/cactus-compute/needle) transformer stack. This mechanism allows distinct information pathways to coexist within the same model depth, controlled by a configurable hyper-parameter. Below is a complete breakdown of how the implementation works, from configuration through to the forward pass.

---

## Configuring Multi-Lane Hyper-Connections

The multi-lane behavior is governed by a single configuration field in `TransformerConfig`.

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), line 76:

```python
@dataclass
class TransformerConfig:
    # ... other fields ...

    mhc_lanes: int = 4               # ↩︎ [config line](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L76)

```

- **Default value**: 4 lanes
- **Parameter name**: `mhc_lanes`
- **Trade-off**: More lanes increase expressivity but add parameters through lane-specific `phi_*` matrices

This value determines how many parallel pathways exist through the stack and directly shapes the dimensions of all hyper-connection tensors.

---

## Building the Lane Matrix in Stack Initialization

The `Stack` class prepares per-layer lane assignments during its `__call__` method. This happens once, before the scanned loop begins execution.

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), lines 93-95:

```python
n, L, nC = cfg.mhc_lanes, cfg.num_layers, cfg.mhc_lanes * cfg.d_model
lane = np.eye(n, dtype=np.float32)[np.arange(L) % n]   # Shape: (L, n)

```

The `lane` tensor works as follows:

- Each layer index `ℓ` is assigned to lane `ℓ % mhc_lanes`
- The result is a one-hot encoding: layer 0 → lane 0, layer 1 → lane 1, ..., layer n → lane 0 again
- This cyclic assignment ensures even distribution across the stack depth

### Lane-Specific Offset Tensors

The configuration builds three offset tensors that bias each lane's gates differently:

```python
hc = {
    "phi_pre":  self.param("mhc_phi_pre", nn.initializers.normal(), (L, nC, n)),
    "phi_post": self.param("mhc_phi_post", nn.initializers.normal(), (L, nC, n)),
    "phi_res":  self.param("mhc_phi_res", nn.initializers.normal(), (L, nC, n, n)),
    "pre_off":  jnp.asarray(8 * lane - 4),               # Range: [-4, +4]

    "post_off": jnp.asarray(-4 * (1 - lane)),           # Lane 0: 0, others: -4

    # ... learned a_* and b_* parameters ...

}

```

These offsets (`pre_off`, `post_off`) inject inductive biases that prevent lane collapse and encourage differentiated behavior across pathways.

---

## Hyper-Connection Computation in the Forward Pass

The actual gating occurs inside `_ScanBody.__call__` during each layer's computation. This scanned body receives the `hc` dictionary and applies three distinct mixing operations.

### Pre-Mix Gate (Before Attention)

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), lines 61-63:

```python
hpre = nn.sigmoid(
    hc["a_pre"] * (nx @ hc["phi_pre"].astype(jnp.float32))
    + hc["b_pre"] + hc["pre_off"]
)

```

- **Purpose**: Modulate the input before the attention/FFN block
- **Lane-specificity**: `phi_pre` has shape `(L, nC, n)` — each layer learns its own projection
- **Offset effect**: `pre_off` ranges from -4 to +4 depending on lane assignment

### Post-Mix Gate (After Attention)

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), lines 66-68:

```python
hpost = 2 * nn.sigmoid(
    hc["a_post"] * (nx @ hc["phi_post"].astype(jnp.float32))
    + hc["b_post"] + hc["post_off"]
)

```

- **Output scaling**: The factor of 2 expands the gate's dynamic range
- **Lane differentiation**: `post_off` is 0 for lane 0, -4 for all other lanes

### Residual-Mix with Sinkhorn Normalization

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), lines 69-71:

```python
res  = nx @ hc["phi_res"].astype(jnp.float32)
hres = _sinkhorn(hc["a_res"] * res.reshape(B, T, n, n) + hc["b_res"])

```

- **`phi_res`**: Projects to a 4-D tensor reshaped as `(batch, time, lanes, lanes)`
- **`_sinkhorn`**: Applies iterative row/column normalization to produce a doubly stochastic mixing matrix
- **Purpose**: Enables structured cross-lane information exchange

### Final State Combination

The three gates combine with the block output `y` and transformed input `xf`:

```python
new_x = (jnp.einsum("btij,btjc->btic", hres, xf)
         + hpost[..., None] * y.astype(jnp.float32)[:, :, None, :]).astype(self.dtype)

```

- **`hres`**: Mixes across lanes via Einstein summation
- **`hpost`**: Scales the attention block output per lane
- **Result**: Updated hidden state with shape `(B, T, mhc_lanes, d_model/mhc_lanes)` or fused equivalent

---

## Practical Implementation Examples

### Enabling Custom Lane Counts

```python
from needle.model.architecture import TransformerConfig, Stack
import jax.numpy as jnp

# Configure 6 lanes for a 12-layer model

cfg = TransformerConfig(
    mhc_lanes=6,        # Override default 4

    num_layers=12,
    d_model=512,
    num_heads=8,
    num_kv_heads=4
)
stack = Stack(cfg)

# Forward pass with multi-lane hyper-connections active

x = jnp.zeros((1, 4, cfg.d_model))
out, _ = stack(x)      # Internal shape: lanes fold into d_model

```

### Inspecting Learned Hyper-Connection Parameters

```python
import jax

# Initialize and capture parameters

params = stack.init(jax.random.PRNGKey(0), x)[1]

# Access lane-specific projection matrices

phi_pre = params['Stack_0']['layers']['mhc_phi_pre']   # Shape: (12, 3072, 6)

phi_post = params['Stack_0']['layers']['mhc_phi_post']  # Shape: (12, 3072, 6)

phi_res = params['Stack_0']['layers']['mhc_phi_res']   # Shape: (12, 3072, 6, 6)

# Each layer's pre-mix projection: phi_pre[layer_idx]  # (3072, 6)

```

---

## Multi-Lane Hyper-Connections vs. Standard Residual Connections

| Aspect | Standard Transformer | Needle 2 with MHC |
|--------|----------------------|-------------------|
| **Information pathways** | Single sequential stream | `mhc_lanes` parallel streams |
| **Cross-layer mixing** | Only via residual addition | Learned `phi_*` matrices + Sinkhorn |
| **Per-layer parameters** | Shared or independent | Lane-cycled with offsets |
| **Computational overhead** | Baseline | ~3× small matmuls per layer |

The key insight from the [cactus-compute/needle](https://github.com/cactus-compute/needle) source is that hyper-connections add structured diversity without requiring deeper networks—information can travel through lane-specific "shortcuts" that standard residual connections cannot model.

---

## Summary

- **`mhc_lanes`** in `TransformerConfig` controls parallelism (default 4)
- The **lane matrix** cyclically assigns layers to lanes via modulo arithmetic
- Three **gating mechanisms** (`hpre`, `hpost`, `hres`) operate with lane-specific projections
- **Sinkhorn normalization** enforces structured cross-lane mixing in the residual path
- All implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py): `Stack.__call__` for setup, `_ScanBody.__call__` for execution

---

## Frequently Asked Questions

### How does the lane assignment work across layers?

Layers are assigned to lanes using cyclic indexing: layer `ℓ` belongs to lane `ℓ % mhc_lanes`. This creates balanced distribution—layer 0 and layer `n` share lane 0, layer 1 and layer `n+1` share lane 1, etc. The `lane` matrix in `Stack.__call__` encodes this as one-hot vectors that index into bias offsets.

### What is the purpose of `pre_off` and `post_off` in the gating computation?

These offsets break symmetry between lanes. `pre_off` spans [-4, +4] based on lane index, while `post_off` is 0 for lane 0 and -4 for others. Without these, all lanes could converge to functionally identical behavior; the biases encourage specialized roles for each pathway.

### Why does the residual gate use Sinkhorn normalization?

The `_sinkhorn` function produces doubly stochastic matrices, meaning each lane's output is a weighted combination of all lane inputs where weights sum to 1. This ensures stable gradient flow and prevents any single lane from dominating the mixing operation, unlike softmax which only enforces row-wise constraints.

### Can I change `mhc_lanes` after training a model?

No—`mhc_lanes` is baked into parameter shapes. The `phi_pre`, `phi_post`, and `phi_res` tensors have `mhc_lanes` in their dimensions, as do the offset tensors. Changing this value requires re-initialization. However, you can train with different values and compare the resulting capacity-efficiency trade-offs.