# How the Hadamard MLP Is Implemented in Needle 2: Architecture and Code Walkthrough

> Explore the Hadamard MLP implementation in Needle 2. Discover how it uses Walsh-Hadamard transform for efficient feature mixing without dense matrix multiplications. Get the code walkthrough.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-30

---

**Needle 2 replaces the traditional feed-forward network with a HadamardMLP that leverages the Walsh-Hadamard transform and learnable diagonal scalings to perform fast, orthogonal feature mixing without dense matrix multiplications.**

The HadamardMLP is a core innovation in the Needle 2 transformer architecture from the **cactus-compute/needle** repository. Unlike standard MLPs that rely on expensive dense linear projections, the Hadamard MLP implementation in Needle 2 uses structured matrix operations to reduce computational overhead while maintaining expressive power. This article examines the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) to explain exactly how this efficient feed-forward layer works.

## Core Architecture of HadamardMLP

The `HadamardMLP` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 87-103) reimagines the feed-forward block as a sequence of diagonal scalings interleaved with Walsh-Hadamard transforms.

### Power-of-Two Padding and Walsh-Hadamard Matrix Generation

The implementation first rounds the input dimension `d_model` up to the nearest power of two, denoted as `n`. It then generates a Walsh-Hadamard matrix `H` of size `n × n` via the internal `_walsh_matrix` function, caching this as a constant tensor converted to the appropriate `dtype` (`H = _walsh_matrix(n).astype(self.dtype)`). This orthogonal matrix enables the **O(n log n)** fast Walsh-Hadamard transform instead of the **O(n²)** cost of dense matrix multiplication.

### Learnable Diagonal Scaling Vectors

Three trainable parameters—**`d1`**, **`d2`**, and **`d3`**—are initialized as diagonal vectors of length `n`. According to the source code, `d1` and `d2` use `jax.nn.initializers.ones`, while `d3` initializes to a constant value of `0.02`. These vectors act as per-feature gain and bias terms applied before and after the transform operations, allowing the model to learn feature-wise scaling while preserving the orthogonal structure of the Hadamard transform.

### Two-Stage Hadamard Mixing Forward Pass

The forward pass executes a three-step mixing process:

1. **First Transform:** The padded input is element-wise multiplied by `d1`, then matrix-multiplied with the Walsh-Hadamard matrix `H` (`z = (d1 * padded_x) @ H`).
2. **Non-linearity and Second Transform:** The result is scaled by `d2`, passed through a **SiLU** activation, and transformed again by `H` (`z = nn.silu(d2 * z) @ H`).
3. **Final Scaling:** The output is scaled by `d3` and truncated back to the original `d_model` dimension (`output = (d3 * z)[..., : d_model]`).

## Integration into the Transformer Block

Inside the `Block` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the HadamardMLP is invoked immediately after the self-attention mechanism. The code passes the token tensor through `HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)` and adds the result to the residual stream via `skip + x`. This placement follows the standard transformer pattern but substitutes the Hadamard-based feed-forward path for the traditional dense MLP.

## Implementation Details in needle/model/architecture.py

The complete implementation spans lines 87-103 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), where the `HadamardMLP` class encapsulates the logic. The `_walsh_matrix` helper function generates the binary-valued orthogonal matrix required for the transform. The `Block` class (lines 34-38) demonstrates the integration point, while `SimpleAttentionNetwork` (lines 78-86) constructs the full model stack that utilizes this architecture.

## Practical Usage Examples

### Standalone HadamardMLP

```python
import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# Dummy batch: (batch, seq_len, d_model)

x = jnp.ones((2, 16, 512), dtype=jnp.bfloat16)

# Initialise the module

hadamard = HadamardMLP(d_model=512, dtype=jnp.bfloat16)

# Initialise parameters with a random key

variables = hadamard.init(jax.random.PRNGKey(0), x)

# Apply the Hadamard‑MLP

y = hadamard.apply(variables, x)
print(y.shape)          # → (2, 16, 512)

```

### Full Transformer with HadamardMLP

```python
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(d_model=512, num_heads=8, num_layers=12)
net = SimpleAttentionNetwork(config=cfg)

# Dummy token IDs

tokens = jnp.arange(0, 64).reshape(1, 64)  # shape (batch, seq_len)

# Initialise & run

params = net.init(jax.random.PRNGKey(0), tokens)
logits = net.apply(params, tokens)  # logits = attention + HadamardMLP output

```

## Summary

- The **HadamardMLP** replaces dense weight matrices with the Walsh-Hadamard transform, reducing computational complexity from O(n²) to O(n log n) for the mixing operations.
- Three learnable diagonal vectors (`d1`, `d2`, `d3`) provide trainable per-feature scaling while maintaining orthogonal mixing properties.
- The implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 87-103) and integrates into the transformer via the `Block` class after the attention mechanism.
- The layer automatically handles power-of-two padding and truncation to accommodate arbitrary `d_model` dimensions.

## Frequently Asked Questions

### What is a Hadamard MLP and why does Needle 2 use it?

A Hadamard MLP is a feed-forward neural network layer that uses the Walsh-Hadamard transform instead of dense matrix multiplication to mix features across the hidden dimension. Needle 2 uses this architecture to reduce computational cost while maintaining the expressiveness of traditional MLPs through learnable diagonal scalings.

### How does the Walsh-Hadamard transform improve efficiency over standard MLPs?

The Walsh-Hadamard transform can be computed in O(n log n) time using fast matrix operations, whereas standard MLP dense layers require O(n²) operations for matrix multiplication. This reduces the computational bottleneck in the feed-forward layers while preserving the orthogonal properties needed for stable gradient flow.

### Where is the HadamardMLP class defined in the Needle repository?

The `HadamardMLP` class is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) between lines 87 and 103. This file also contains the `_walsh_matrix` helper function and the `Block` class that integrates HadamardMLP into the complete transformer architecture.

### Can I use HadamardMLP as a standalone module outside the Transformer?

Yes, the `HadamardMLP` class is a standard Flax module that can be instantiated independently with `HadamardMLP(d_model=512, dtype=jnp.bfloat16)`. It accepts tensors of shape `(batch, seq_len, d_model)` and can be used in any JAX-based architecture requiring efficient feature mixing.