# Multi-Lane Hyper-Connections in Needle 2: Architecture and Implementation

> Discover Needle 2's multi-lane hyper-connections, learned transformations that dynamically route outputs between parallel computational lanes for efficient information sharing.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-09-09

---

**Multi-lane hyper-connections in Needle 2 are learned transformations that route outputs between parallel computational lanes—attention, engram memory, and Hadamard MLP—enabling dynamic information sharing with minimal parameter overhead.**

Needle 2 introduces a novel "Simple Attention Network" architecture that replaces traditional transformer components with specialized lanes interconnected through multi-lane hyper-connections. According to the project README, this design replaces the feed-forward network with a Hadamard MLP, uses Grouped-Query Attention (GQA), adds an engram key-value memory, and connects these components through adaptive hyper-connections. This article explains how these connections work and how they are implemented in the `cactus-compute/needle` repository.

## What Are Multi-Lane Hyper-Connections?

A **hyper-connection** is a learned transformation that routes the output of one "lane" (a sub-module such as an attention block or key-value memory) into the input of another lane. Unlike a single linear projection shared across all paths, each lane maintains its own **hyper-network** parameters that can adaptively weight and blend information coming from several parallel streams.

This architecture allows the model to bypass the usual sequential stack. Instead of processing attention, then memory, then MLP in strict order, Needle 2 processes these lanes in parallel and uses hyper-connections to inject the results of one lane directly into another.

## The Three Computational Lanes in Needle 2

The multi-lane architecture consists of three primary lanes that operate simultaneously during the forward pass:

### Attention Lane with GQA

The attention lane implements **Grouped-Query Attention (GQA)**, which shares key and value heads across multiple query heads to reduce memory bandwidth while maintaining model quality. This lane processes the input sequence through the attention mechanism to capture contextual relationships.

### Engram Key-Value Memory

The **engram key-value memory** lane provides an external memory mechanism that stores and retrieves information outside the standard attention context window. This allows the model to access persistent information across layers without increasing the attention computation cost.

### Hadamard MLP Lane

Replacing the traditional feed-forward network (FFN), the **Hadamard MLP** lane uses element-wise multiplication (Hadamard product) operations instead of standard matrix multiplications. This reduces computational complexity while maintaining expressive capacity.

## Benefits of the Hyper-Connection Design

Multi-lane hyper-connections provide four distinct advantages over standard transformer architectures:

- **Higher capacity with low overhead**: Each lane adds only a modest number of parameters (the hyper-network weights) but dramatically expands the expressive power of the model by enabling complex interactions between lanes.

- **Cross-lane information sharing**: Hyper-connections allow the result of one lane (e.g., the engram memory) to be injected directly into another lane (e.g., the Hadamard-MLP), bypassing the usual sequential stack and enabling richer feature representations.

- **Dynamic routing**: Because the hyper-network is conditioned on the current hidden state, the model can decide which lanes to emphasize for a given token, enabling context-dependent computation paths.

- **Improved gradient flow**: Direct shortcuts between lanes reduce the depth that gradients must traverse during backpropagation, helping training stability especially for the smaller "dense" models targeted by Needle 2.

## Implementation in needle/model/architecture.py

The core implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), where the model's forward pass builds a set of parallel sub-components and merges them through learned hyper-projectors. The architecture defines separate attributes for each lane:

- `model.attention` for the GQAttention lane
- `model.engram` for the key-value memory
- `model.hadamard` for the Hadamard MLP
- `model.hyper_proj` for the hyper-connection weights (shape: `(num_lanes, hidden_dim)`)

During the forward pass defined in the architecture module, these lanes execute in parallel and their outputs are combined through the hyper-projection matrices. The file [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) demonstrates how this forward pass invokes the architecture, wiring the hyper-connections into the generation pipeline.

## Working with Multi-Lane Hyper-Connections in Code

While hyper-connections operate internally, you can instantiate and interact with Needle 2 as follows:

```python
from needle import Needle, tool

# Create a Needle 2 instance

agent = Needle(tools=[])

# Simple generation

response = agent.run("Explain the benefits of multi-lane hyper-connections.")
print(response)

```

For debugging or research purposes, you can inspect the internal lane structure and hyper-connection parameters:

```python

# Access the underlying model architecture

model = agent._lib  # internal handle to the compiled library

# The architecture module defines the lanes:

attention_lane = model.attention
engram_memory = model.engram
hadamard_mlp = model.hadamard

# Hyper-projectors (the learned connections) are stored as parameters:

hyper_proj = model.hyper_proj  # shape: (num_lanes, hidden_dim)

```

## Summary

- **Multi-lane hyper-connections** are learned routing mechanisms that connect parallel computational lanes in Needle 2's architecture.
- The architecture consists of three lanes: **GQA Attention**, **Engram Key-Value Memory**, and **Hadamard MLP**.
- Hyper-connections enable **cross-lane information sharing** and **dynamic routing** without the parameter cost of full layer projections.
- Implementation is centralized in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), with hyper-projection weights stored in `model.hyper_proj`.
- These connections improve gradient flow and allow Needle 2 to achieve strong performance with a compact model footprint.

## Frequently Asked Questions

### What is a "lane" in Needle 2's architecture?

A **lane** refers to a distinct computational sub-module within the architecture—specifically the attention mechanism, the engram memory, or the Hadamard MLP. Unlike traditional transformers that process components sequentially, Needle 2 processes these lanes in parallel and uses hyper-connections to blend their outputs.

### How do hyper-connections differ from standard residual connections?

While residual connections add a sub-layer's output to its input (creating skip gradients), hyper-connections are **learned transformations** that weight and route information between different lanes. They allow the attention lane to directly influence the Hadamard MLP lane, for example, with adaptive weights generated by a hyper-network conditioned on the current hidden state.

### Where are the hyper-connection weights stored in the model?

The learned hyper-projectors are stored as model parameters accessible via `model.hyper_proj` in the architecture module, with a shape of `(num_lanes, hidden_dim)`. These weights are defined and initialized in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and updated during training alongside the other lane parameters.

### Why does Needle 2 use a Hadamard MLP instead of a standard FFN?

The **Hadamard MLP** replaces matrix multiplications with element-wise Hadamard products, reducing computational overhead while maintaining sufficient model capacity. When combined with multi-lane hyper-connections, this allows the model to achieve competitive performance with fewer total parameters compared to traditional Feed-Forward Networks.