# How the Hadamard MLP Differs from Standard Transformer FFN Layers

> Discover how the Hadamard MLP revolutionizes transformer FFN layers by replacing dense layers with a Walsh-Hadamard transform, drastically reducing parameters while preserving performance. Learn more.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-09

---

**The Hadamard MLP replaces the two dense layers of a standard transformer feed-forward network with a Walsh-Hadamard transform sandwiched between three learnable diagonal matrices, reducing trainable parameters from quadratic to linear complexity while maintaining expressive power.**

The Needle repository introduces a novel feed-forward architecture that challenges the conventional transformer design. Unlike standard FFN layers that rely on two dense matrix multiplications, the **Hadamard MLP** leverages fixed orthogonal transforms and diagonal scaling to achieve dramatic efficiency gains. This architectural shift affects everything from parameter counts to memory footprints and computational complexity.

## Architectural Structure: Dense Layers vs. Hadamard Transforms

Standard transformer FFN layers follow a well-established pattern of matrix multiplications separated by non-linearities. The Hadamard MLP fundamentally reimagines this computation using orthogonal basis transforms.

### Standard Transformer FFN Design

A conventional feed-forward network in transformers consists of two dense linear layers with an activation function between them. The forward pass follows the pattern `Dense → Activation → Dense`, typically using GELU or ReLU activations. This structure requires storing full weight matrices of size `d_model × d_ff` for both layers, resulting in a parameter count of `2·d_model·d_ff` plus biases. When `d_ff` is set to `4·d_model` (the common default), this creates quadratic complexity `O(d_model²)` for both storage and computation.

### Hadamard MLP Structure

In contrast, the Hadamard MLP implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 87-103) replaces these dense projections with a **Walsh-Hadamard transform** (`H`) enclosed by three learnable diagonal matrices (`d1`, `d2`, `d3`). The forward pass executes as `z = (d1·x)·H → silu(d2·z)·H → d3·z`, where the SiLU activation applies element-wise between the two Hadamard operations. The Hadamard matrix itself is generated on-the-fly by the `_walsh_matrix` function (lines 80-84) and requires no training, as it remains fixed during optimization.

## Parameter Efficiency and Computational Complexity

The most striking difference between these architectures lies in their scaling characteristics. While standard FFN layers scale quadratically with model dimension, the Hadamard MLP maintains linear scaling.

Standard FFN parameters grow as `2·d_model·d_ff`, which becomes prohibitively large for high-dimensional models. The Hadamard MLP stores only **three vectors of length `n`** (where `n` is the next power-of-two greater than or equal to `d_model`), resulting in approximately `3·d_model` trainable parameters. This represents a reduction from quadratic to linear complexity relative to the hidden dimension.

Computationally, standard dense layers require `O(d_model·d_ff)` operations, typically `O(d_model²)` when using the 4× expansion ratio. The Hadamard MLP leverages the fast Hadamard algorithm, which operates in `O(n·log n)` time. Combined with the efficient diagonal matrix multiplications, the overall complexity stays close to `O(d_model·log d_model)`, offering significant speedups for large hidden dimensions.

## Memory Footprint and Hardware Optimization

Memory constraints differentiate these architectures significantly in production environments. Standard FFN implementations must persist full weight matrices for both dense layers in GPU memory, creating bandwidth bottlenecks during training and inference.

The Hadamard MLP eliminates this storage overhead by generating the Walsh-Hadamard matrix computationally rather than storing it as parameters. Only the three diagonal scaling vectors occupy memory, drastically reducing the model's footprint. This design proves particularly advantageous for deployment scenarios where memory bandwidth limits throughput, as the reduced parameter count enables larger batch sizes and more efficient caching.

## Integration Within Transformer Blocks

Placement and normalization strategies further distinguish these architectures within the transformer stack.

Standard transformer blocks typically sequence operations as `x → Attn → Add → Norm → FFN → Add`, applying layer normalization or RMSNorm before the FFN. Needle's `Block` class (lines 305-337) modifies this flow to `x → Attn → Add → Norm → HadamardMLP → Add`, utilizing **ZCRMSNorm** (also referenced as `pre_hada_norm`) before the Hadamard MLP sub-layer. This normalization strategy complements the orthogonal properties of the Hadamard transform, ensuring stable gradients across the diagonal scaling operations.

## Implementation Details in Needle

Examining the source code reveals the precise implementation mechanics. The `HadamardMLP` class defines three parameters via `nn.param`: `d1`, `d2`, and `d3`, each initialized as diagonal scaling vectors. The forward method applies these scalings interleaved with matrix multiplication against the pre-computed Walsh-Hadamard matrix, with SiLU providing the necessary non-linearity between transforms.

The top-level `SimpleAttentionNetwork` (lines 78-86) stacks these blocks to create the complete model, automatically integrating the Hadamard MLP into each residual block without requiring explicit configuration changes from the user perspective.

## Usage Examples

Instantiating a model with Hadamard MLP layers requires no special syntax beyond standard Needle configuration:

```python
import jax.numpy as jnp
import flax.linen as nn
from needle.model.architecture import Block, SimpleAttentionNetwork, TransformerConfig

# Configure transformer parameters

cfg = TransformerConfig(d_model=512, num_heads=8, num_kv_heads=4, num_layers=4)

# Build the model - Block automatically uses HadamardMLP

model = SimpleAttentionNetwork(cfg)

# Prepare input batch

batch_size = 2
seq_len = 16
tokens = jnp.ones((batch_size, seq_len), dtype=jnp.int32)

# Execute forward pass

logits = model(tokens)  # Shape: (batch, seq_len, vocab_size)

```

To inspect the Hadamard MLP component directly:

```python
from needle.model.architecture import HadamardMLP

# Create sample input

x = jnp.ones((2, 10, 512))
hada = HadamardMLP(d_model=512)

# Process through Hadamard transform and diagonal scalings

y = hada(x)  # Output shape matches input: (2, 10, 512)

```

## Summary

- The **Hadamard MLP** replaces dual dense layers with a Walsh-Hadamard transform and three diagonal matrices, reducing parameters from `O(d_model²)` to `O(d_model)`.
- Computational complexity improves from quadratic to `O(d_model·log d_model)` through the fast Hadamard algorithm.
- Memory efficiency increases substantially since the orthogonal transform matrix requires no storage, unlike dense weight matrices.
- The architecture is implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and integrates seamlessly into the standard transformer block structure with specialized ZCRMSNorm normalization.

## Frequently Asked Questions

### What is the Walsh-Hadamard transform and why does the Hadamard MLP use it?

The Walsh-Hadamard transform is an orthogonal, linear transformation that operates similarly to a Fourier transform but uses square wave functions (Walsh functions) as its basis. The Hadamard MLP utilizes this transform because it provides a fixed, invertible mixing of input features without requiring trainable parameters. According to the Needle source code, this orthogonal basis encourages uniform feature mixing while the three learnable diagonal matrices (`d1`, `d2`, `d3`) provide the necessary expressive capacity for task-specific transformations.

### How much memory does the Hadamard MLP save compared to a standard FFN?

The Hadamard MLP stores only three vectors of length equal to the model dimension (rounded to the next power of two), compared to two full weight matrices of size `d_model × d_ff` in standard architectures. For a typical configuration where `d_ff = 4·d_model`, this reduces the parameter count from approximately `8·d_model²` to `3·d_model`. This represents roughly a **97% reduction** in FFN-specific parameters for a 512-dimensional model, translating directly to reduced GPU memory requirements and smaller checkpoint sizes.

### Can I use the Hadamard MLP with pre-trained transformer weights?

No, the Hadamard MLP is not directly compatible with standard transformer checkpoints due to its fundamentally different parameter structure. While standard FFNs contain dense weight matrices, the Hadamard MLP relies on three diagonal scaling vectors interleaved with fixed orthogonal transforms. Conversion would require retraining or specialized distillation techniques, as the architectural invariants differ significantly between the two approaches.

### Why is the Hadamard MLP faster on modern hardware despite using matrix multiplications?

Although both architectures utilize matrix operations, the Hadamard MLP's fixed Walsh-Hadamard matrix possesses special structure that allows optimized implementation via the fast Hadamard algorithm, similar to the fast Fourier transform. This reduces complexity from `O(n²)` to `O(n·log n)`. Additionally, the three diagonal matrix multiplications are highly parallelizable and memory-efficient compared to general dense matrix multiplications, reducing memory bandwidth bottlenecks that often limit GPU utilization in standard transformer FFN layers.