# How the Hadamard MLP Layer Works in Needle 2: A Deep Dive into Parameter-Efficient Feed-Forward Networks

> Explore the Hadamard MLP layer in Needle 2. Discover how this parameter-efficient feed-forward network uses diagonal vectors instead of dense matrices for efficient feature mixing.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-01

---

**The Hadamard MLP layer in Needle 2 replaces conventional two-layer feed-forward networks with a Walsh-Hadamard transform-based architecture that uses only three learned diagonal vectors instead of dense weight matrices, reducing parameters while maintaining dense feature mixing.**

The **Hadamard MLP** is a compact feed-forward module used in Needle 2's transformer blocks. It reimagines the standard `Dense → activation → Dense` stack by leveraging the mathematical properties of the Walsh-Hadamard transform to achieve efficient feature mixing. This architecture, implemented in the open-source [cactus-compute/needle](https://github.com/cactus-compute/needle) repository, demonstrates how deterministic orthogonal transforms can replace learned linear projections without sacrificing model capacity.

## Hadamard MLP Architecture Overview

The Hadamard MLP layer is defined in [[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Unlike conventional MLPs that learn full weight matrices, this design relies on fixed orthogonal transforms combined with minimal learned scaling.

The core insight: multiply by a **Walsh-Hadamard matrix** `H` spreads information across all dimensions without adding trainable parameters. Three small diagonal vectors—`d1`, `d2`, and `d3`—provide the only learned capacity, making this approach dramatically more parameter-efficient.

## Step-by-Step Implementation in Needle 2

### Step 1: Generating the Walsh-Hadamard Matrix

The [`_walsh_matrix(n)`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L80-L84) function constructs an `n×n` orthogonal matrix whose rows are Walsh functions:

```python

# From needle/model/architecture.py, lines 80-84

def _walsh_matrix(n: int) -> jnp.ndarray:
    # Build recursively using Sylvester's construction

    # Returns H normalized by sqrt(n) so that H @ H.T = I

```

This matrix is **deterministic**—no training required—and satisfies `H·Hᵀ = I` due to proper normalization by `√n`.

### Step 2: Input Padding to Power-of-Two

Hadamard transforms require dimensions that are powers of two. The [padding logic](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L98-L100) handles this automatically:

- Input tensor shape: `(B, T, d_model)` where `B` = batch, `T` = sequence length
- If `d_model` is not a power of two, zero-pad to next power-of-two `n`

This padding is sliced away at the output, preserving the original `d_model` dimension.

### Step 3-7: The Forward Pass Pipeline

| Stage | Operation | Parameters | Source Location |
|-------|-----------|------------|-----------------|
| **Pre-transform scaling** | `z = d1 * x` | Learned diagonal `d1` (init=1) | [lines 95-96](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L95-L96) |
| **First Hadamard** | `z = z @ H` | None (fixed matrix) | [line 100](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L100) |
| **Gating** | `z = d2 * z` | Learned diagonal `d2` (init=1) | [line 101](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L101) |
| **Activation** | `z = SiLU(z)` | None | [line 102](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L102) |
| **Second Hadamard** | `z = z @ H` | None (fixed matrix) | [line 102](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L102) |
| **Output scaling** | `z = d3 * z` | Learned diagonal `d3` (init=0.02) | [lines 102-103](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L102-L103) |
| **Projection** | Slice to `d_model` | None | [line 103](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L103) |

The initialization strategy matters: `d1` and `d2` start at 1.0 (identity-like behavior), while `d3` initializes to 0.02 for training stability.

## Hadamard MLP vs. Conventional MLP: Parameter Comparison

| Component | Conventional MLP | Hadamard MLP |
|-----------|-----------------|--------------|
| First projection | `d_model × d_ff` weights | `d1`: `d_model` values |
| Second projection | `d_ff × d_model` weights | `d3`: `d_model` values |
| Intermediate gating | Bias terms, optional | `d2`: `d_model` values |
| **Total parameters** | `O(d_model × d_ff)` | **`O(d_model)`** |

For typical settings where `d_ff = 4 × d_model`, this reduces parameters from `~8×d_model²` to `~3×d_model`—roughly a **2.6× reduction** in feed-forward parameters alone.

## Using the Hadamard MLP Layer in Practice

### Direct Invocation

```python
import jax
import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# Dummy input: batch=2, seq_len=10, d_model=64

x = jnp.ones((2, 10, 64), dtype=jnp.bfloat16)

# Create the layer (d_model must match input size)

mlp = HadamardMLP(d_model=64, dtype=jnp.bfloat16)

# Apply (inside a JAX/Flax apply call)

params = mlp.init(jax.random.PRNGKey(0), x)   # initialise parameters

y = mlp.apply(params, x)                     # forward pass

print(y.shape)   # → (2, 10, 64)

```

### Inside a Full Transformer Block

The Hadamard MLP integrates into Needle 2's transformer stack through the `Block` class, as shown in the [`Block` forward pass](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L34-L38):

```python
from needle.model.architecture import Block, Stack, TransformerConfig

cfg = TransformerConfig(
    num_heads=8,
    num_kv_heads=8,
    d_model=64,
    num_layers=12,
    jax_dtype=jnp.bfloat16,
    # … other config fields …

)

stack = Stack(cfg)

# Initialise with a dummy input

x = jnp.ones((1, 128, 64), dtype=jnp.bfloat16)
params = stack.init(jax.random.PRNGKey(0), x)
out, hidden = stack.apply(params, x)
print(out.shape)   # → (1, 128, 64)

```

Within each `Block`, the Hadamard MLP follows self-attention with pre-normalization (`ZCRMSNorm`) and a residual connection: `output = skip + mlp(norm(x))`.

## Why the Hadamard Transform Works

### Dense Mixing Without Dense Parameters

Multiplying by `H` provides **all-to-all interaction**: every output dimension depends on every input dimension. This matches the expressive goal of dense linear layers but derives the mixing from a fixed mathematical structure rather than learned weights.

The Walsh-Hadamard matrix is particularly suited for this because:
- **Orthogonality** preserves signal norms: `||Hx|| = ||x||`
- **Fast computation**: The Fast Walsh-Hadamard Transform (FWHT) runs in `O(n log n)` vs. `O(n²)` for naive matrix multiplication
- **Hardware efficiency**: Requires only additions and subtractions—no multiplications in the transform itself

### Training Dynamics

The three diagonal parameters serve distinct roles:
- **`d1`** — Pre-transform scaling: controls how strongly each channel participates in the first mixing
- **`d2`** — Gating: creates channel-wise multiplicative interactions after the first spread
- **`d3`** — Output scaling: learns the final projection strength with conservative initialization (0.02) to prevent gradient explosion early in training

## Integration with Quantization and Export

The Hadamard representation appears in Needle 2's weight export pipeline. In [[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the `H` matrix participates in quantization-aware transformations, allowing the deterministic transform to be fused with efficient integer arithmetic at inference time.

## Summary

- The **Hadamard MLP** in Needle 2 replaces dense weight matrices with the Walsh-Hadamard transform plus three learned diagonal vectors (`d1`, `d2`, `d3`)
- This reduces feed-forward parameters from `O(d_model × d_ff)` to `O(d_model)` while maintaining dense feature mixing through orthogonal transforms
- The implementation in [[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) handles power-of-two padding, SiLU gating, and residual connections automatically
- Fast Walsh-Hadamard algorithms and hardware-friendly operations (additions only) make this practical for production deployment

## Frequently Asked Questions

### What is the Walsh-Hadamard transform used for in neural networks?

The Walsh-Hadamard transform provides a fixed, orthogonal linear mapping that spreads information across all dimensions without learned parameters. In Needle 2's Hadamard MLP, it replaces the role of dense weight matrices in conventional feed-forward networks, enabling dense feature mixing at a fraction of the parameter cost.

### Why does the Hadamard MLP require power-of-two dimensions?

The Fast Walsh-Hadamard Transform (FWHT) algorithm requires dimensions that are powers of two for efficient recursive decomposition. Needle 2 automatically pads inputs to the next power of two and slices the output, hiding this constraint from the caller while preserving the original `d_model` size.

### How does the Hadamard MLP compare to MLP-Mixer or other token-mixing architectures?

Unlike MLP-Mixer, which learns separate token-mixing and channel-mixing MLPs, the Hadamard MLP uses **fixed, deterministic transforms** for spatial mixing while learning only minimal channel-wise scalings. This makes it more parameter-efficient than learned mixing schemes, though with less flexibility per layer.

### Can the Hadamard MLP be used outside transformer architectures?

Yes—the `HadamardMLP` class in Needle 2 is architecture-agnostic. Any scenario requiring a compact feed-forward module can benefit from this approach, particularly resource-constrained settings where parameter efficiency outweighs the need for maximum representational flexibility per layer.