How the Hadamard MLP Layer Works in Needle 2: A Deep Dive into Parameter-Efficient Feed-Forward Networks

The Hadamard MLP layer in Needle 2 replaces conventional two-layer feed-forward networks with a Walsh-Hadamard transform-based architecture that uses only three learned diagonal vectors instead of dense weight matrices, reducing parameters while maintaining dense feature mixing.

The Hadamard MLP is a compact feed-forward module used in Needle 2's transformer blocks. It reimagines the standard Dense → activation → Dense stack by leveraging the mathematical properties of the Walsh-Hadamard transform to achieve efficient feature mixing. This architecture, implemented in the open-source cactus-compute/needle repository, demonstrates how deterministic orthogonal transforms can replace learned linear projections without sacrificing model capacity.

Hadamard MLP Architecture Overview

The Hadamard MLP layer is defined in [needle/model/architecture.py](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Unlike conventional MLPs that learn full weight matrices, this design relies on fixed orthogonal transforms combined with minimal learned scaling.

The core insight: multiply by a Walsh-Hadamard matrix H spreads information across all dimensions without adding trainable parameters. Three small diagonal vectors—d1, d2, and d3—provide the only learned capacity, making this approach dramatically more parameter-efficient.

Step-by-Step Implementation in Needle 2

Step 1: Generating the Walsh-Hadamard Matrix

The _walsh_matrix(n) function constructs an n×n orthogonal matrix whose rows are Walsh functions:

# From needle/model/architecture.py, lines 80-84

def _walsh_matrix(n: int) -> jnp.ndarray:
    # Build recursively using Sylvester's construction

    # Returns H normalized by sqrt(n) so that H @ H.T = I

This matrix is deterministic—no training required—and satisfies H·Hᵀ = I due to proper normalization by √n.

Step 2: Input Padding to Power-of-Two

Hadamard transforms require dimensions that are powers of two. The padding logic handles this automatically:

• Input tensor shape: (B, T, d_model) where B = batch, T = sequence length
• If d_model is not a power of two, zero-pad to next power-of-two n

This padding is sliced away at the output, preserving the original d_model dimension.

Step 3-7: The Forward Pass Pipeline

Stage	Operation	Parameters	Source Location
Pre-transform scaling	z = d1 * x	Learned diagonal d1 (init=1)	lines 95-96
First Hadamard	z = z @ H	None (fixed matrix)	line 100
Gating	z = d2 * z	Learned diagonal d2 (init=1)	line 101
Activation	z = SiLU(z)	None	line 102
Second Hadamard	z = z @ H	None (fixed matrix)	line 102
Output scaling	z = d3 * z	Learned diagonal d3 (init=0.02)	lines 102-103
Projection	Slice to d_model	None	line 103

The initialization strategy matters: d1 and d2 start at 1.0 (identity-like behavior), while d3 initializes to 0.02 for training stability.

Hadamard MLP vs. Conventional MLP: Parameter Comparison

Component	Conventional MLP	Hadamard MLP
First projection	d_model × d_ff weights	d1: d_model values
Second projection	d_ff × d_model weights	d3: d_model values
Intermediate gating	Bias terms, optional	d2: d_model values
Total parameters	O(d_model × d_ff)	O(d_model)

For typical settings where d_ff = 4 × d_model, this reduces parameters from ~8×d_model² to ~3×d_model—roughly a 2.6× reduction in feed-forward parameters alone.

Using the Hadamard MLP Layer in Practice

Direct Invocation

import jax
import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# Dummy input: batch=2, seq_len=10, d_model=64

x = jnp.ones((2, 10, 64), dtype=jnp.bfloat16)

# Create the layer (d_model must match input size)

mlp = HadamardMLP(d_model=64, dtype=jnp.bfloat16)

# Apply (inside a JAX/Flax apply call)

params = mlp.init(jax.random.PRNGKey(0), x)   # initialise parameters

y = mlp.apply(params, x)                     # forward pass

print(y.shape)   # → (2, 10, 64)

Inside a Full Transformer Block

The Hadamard MLP integrates into Needle 2's transformer stack through the Block class, as shown in the Block forward pass:

from needle.model.architecture import Block, Stack, TransformerConfig

cfg = TransformerConfig(
    num_heads=8,
    num_kv_heads=8,
    d_model=64,
    num_layers=12,
    jax_dtype=jnp.bfloat16,
    # … other config fields …

)

stack = Stack(cfg)

# Initialise with a dummy input

x = jnp.ones((1, 128, 64), dtype=jnp.bfloat16)
params = stack.init(jax.random.PRNGKey(0), x)
out, hidden = stack.apply(params, x)
print(out.shape)   # → (1, 128, 64)

Within each Block, the Hadamard MLP follows self-attention with pre-normalization (ZCRMSNorm) and a residual connection: output = skip + mlp(norm(x)).

Why the Hadamard Transform Works

Dense Mixing Without Dense Parameters

Multiplying by H provides all-to-all interaction: every output dimension depends on every input dimension. This matches the expressive goal of dense linear layers but derives the mixing from a fixed mathematical structure rather than learned weights.

The Walsh-Hadamard matrix is particularly suited for this because:

• Orthogonality preserves signal norms: ||Hx|| = ||x||
• Fast computation: The Fast Walsh-Hadamard Transform (FWHT) runs in O(n log n) vs. O(n²) for naive matrix multiplication
• Hardware efficiency: Requires only additions and subtractions—no multiplications in the transform itself

Training Dynamics

The three diagonal parameters serve distinct roles:

• d1 — Pre-transform scaling: controls how strongly each channel participates in the first mixing
• d2 — Gating: creates channel-wise multiplicative interactions after the first spread
• d3 — Output scaling: learns the final projection strength with conservative initialization (0.02) to prevent gradient explosion early in training

Integration with Quantization and Export

The Hadamard representation appears in Needle 2's weight export pipeline. In [needle/model/export.py](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the H matrix participates in quantization-aware transformations, allowing the deterministic transform to be fused with efficient integer arithmetic at inference time.

Summary

• The Hadamard MLP in Needle 2 replaces dense weight matrices with the Walsh-Hadamard transform plus three learned diagonal vectors (d1, d2, d3)
• This reduces feed-forward parameters from O(d_model × d_ff) to O(d_model) while maintaining dense feature mixing through orthogonal transforms
• The implementation in [needle/model/architecture.py](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) handles power-of-two padding, SiLU gating, and residual connections automatically
• Fast Walsh-Hadamard algorithms and hardware-friendly operations (additions only) make this practical for production deployment

Frequently Asked Questions

What is the Walsh-Hadamard transform used for in neural networks?

The Walsh-Hadamard transform provides a fixed, orthogonal linear mapping that spreads information across all dimensions without learned parameters. In Needle 2's Hadamard MLP, it replaces the role of dense weight matrices in conventional feed-forward networks, enabling dense feature mixing at a fraction of the parameter cost.

Why does the Hadamard MLP require power-of-two dimensions?

The Fast Walsh-Hadamard Transform (FWHT) algorithm requires dimensions that are powers of two for efficient recursive decomposition. Needle 2 automatically pads inputs to the next power of two and slices the output, hiding this constraint from the caller while preserving the original d_model size.

How does the Hadamard MLP compare to MLP-Mixer or other token-mixing architectures?

Unlike MLP-Mixer, which learns separate token-mixing and channel-mixing MLPs, the Hadamard MLP uses fixed, deterministic transforms for spatial mixing while learning only minimal channel-wise scalings. This makes it more parameter-efficient than learned mixing schemes, though with less flexibility per layer.

Can the Hadamard MLP be used outside transformer architectures?

Yes—the HadamardMLP class in Needle 2 is architecture-agnostic. Any scenario requiring a compact feed-forward module can benefit from this approach, particularly resource-constrained settings where parameter efficiency outweighs the need for maximum representational flexibility per layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}