# What Is a Hadamard MLP and How Does It Differ From Traditional Feed‑Forward Networks?

> Discover Hadamard MLPs, a parameter-efficient neural network layer. Learn how they reduce parameters from O(n²) to O(n) using the Walsh-Hadamard transform, offering a powerful alternative to traditional feedforward networks.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-11

---

**A Hadamard MLP is a parameter‑efficient feed‑forward layer that replaces dense weight matrices with three learnable diagonal scaling vectors and the fixed Walsh‑Hadamard transform, reducing trainable parameters from O(n²) to O(n) while maintaining expressiveness through structured orthogonal mixing.**

The **Hadamard MLP** is a specialized component in the **Needle** transformer model developed by `cactus-compute/needle`. Unlike conventional feed‑forward networks that rely on arbitrary dense linear projections, this layer exploits the orthogonal structure of the Walsh‑Hadamard matrix to achieve dramatic memory and compute savings. Understanding its implementation reveals how structured sparsity can replace brute‑force parameter counts in modern transformer architectures.

## Core Architecture and Design

The fundamental innovation of the Hadamard MLP lies in its decomposition of the weight matrix. Rather than learning a full dense matrix **W** of size `d_model × d_model`, the layer learns only three diagonal scaling vectors—**`d1`**, **`d2`**, and **`d3`**—and applies the fixed **Walsh‑Hadamard transform** (**`H`**) repeatedly to mix information across the hidden dimension.

The transform matrix **H** is orthogonal and defined recursively, allowing multiplication in `O(n log n)` time via the fast Walsh‑Hadamard transform (FWHT). In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the helper function `_walsh_matrix` generates this normalized matrix by padding the input dimension to the next power of two (`n`), ensuring efficient transform operations on hardware accelerators.

## Implementation Details in Needle

According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 87‑103), the `HadamardMLP` class implements the following data flow:

1. **Padding**: The input dimension (`d_model`) is padded to the next power‑of‑two `n` to accommodate the Walsh‑Hadamard matrix requirements.
2. **First mixing**: The input is scaled by `d1` and multiplied by the Hadamard matrix **H**.
3. **Non‑linearity**: The result is scaled by `d2` and passed through the SiLU activation (`nn.silu`).
4. **Second mixing**: Another Hadamard transform is applied.
5. **Output projection**: The final vector `d3` scales the output, which is then truncated back to the original `d_model` dimension.

```python

# Conceptual data flow inside HadamardMLP.__call__

z = (d1 * padded_x) @ H            # Linear mixing via Hadamard

z = nn.silu(d2 * z) @ H            # Non-linearity + second mixing

out = (d3 * z)[..., :d_model]      # Final scaling and truncation

```

The three diagonal vectors are defined as learnable parameters using Flax’s `self.param()` convention, appearing as `('d1',)`, `('d2',)`, and `('d3',)` in the parameter tree. This contrasts sharply with standard dense layers that instantiate full weight matrices and bias vectors.

## Comparison: Hadamard MLP vs. Traditional Feed‑Forward Networks

**Traditional MLP layers** compute `y = activation(W·x + b)`, where **W** is a dense matrix requiring `O(d_model²)` parameters and `O(d_model²)` floating‑point operations per token. The **Hadamard MLP** introduces three critical differences:

*   **Parameter Efficiency**: Only `3 × n` scalars are learned versus `n²` dense weights. For a typical `d_model` of 768 (padded to `n = 1024`), this reduces parameters from ~590K to ~3K in the feed‑forward component.
*   **Computational Complexity**: Multiplication by **H** uses the fast Walsh‑Hadamard transform, requiring `O(n log n)` operations instead of the `O(n²)` matrix‑vector product.
*   **Structural Constraint**: The effective weight matrix is implicitly `D3 · H · D2 · H · D1`, where **D** represents diagonal matrices. This enforces a specific low‑rank structure that acts as a strong regularizer, preventing overfitting compared to arbitrary dense configurations.

## Practical Usage Examples

### Instantiating a Stand‑alone Hadamard MLP

For debugging or experimental architectures, you can import and call the layer directly from `needle.model.architecture`:

```python
import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# d_model = 768 (typical for small transformers)

mlp = HadamardMLP(d_model=768, dtype=jnp.bfloat16)

# Dummy input: (batch, seq_len, d_model)

x = jnp.ones((2, 10, 768), dtype=jnp.bfloat16)
y = mlp(x)          # → (2, 10, 768)

```

### Integration Within a Full Transformer Stack

In production usage, the `HadamardMLP` is automatically invoked within the `Block` class (lines 34‑38 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) when constructing the model:

```python
from needle.model.architecture import Stack, TransformerConfig

cfg = TransformerConfig(
    num_layers=12,
    d_model=768,
    num_heads=12,
    num_kv_heads=12,
)

# Build full model; HadamardMLP is used inside each Block

model = Stack(config=cfg)

inputs = jnp.ones((1, 20, cfg.d_model), dtype=jnp.bfloat16)
logits, _ = model(inputs)   # logits shape: (1, 20, cfg.d_model)

```

### Manual Replacement of Standard FFN

To substitute a traditional feed‑forward network with the Hadamard variant in a custom `Block` subclass:

```python
from needle.model.architecture import Block

class SimpleBlock(Block):
    @nn.compact
    def __call__(self, x, **kw):
        # Run attention from parent class

        x = super().__call__(x, **kw)
        # Replace standard MLP with Hadamard version

        x = HadamardMLP(self.d_model, self.dtype)(x)
        return x

```

## Summary

*   The **Hadamard MLP** reduces feed‑forward parameters from quadratic to linear complexity by learning only three diagonal vectors (`d1`, `d2`, `d3`) instead of dense matrices.
*   It utilizes the **Walsh‑Hadamard transform** (`_walsh_matrix` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)) to achieve `O(n log n)` mixing operations via the fast transform algorithm.
*   The layer is implemented in **Flax** and integrated into the Needle transformer stack at `Block` level (lines 34‑38).
*   **Trade‑offs**: Significant memory savings and regularization benefits versus the expressiveness of unconstrained dense projections.

## Frequently Asked Questions

### Why pad to the next power of two?

The Walsh‑Hadamard transform requires matrix dimensions that are powers of two to maintain its recursive, balanced structure. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the implementation pads `d_model` to `n` (the next power of two) and truncates the output back to the original dimension after the final scaling step, ensuring computational efficiency without altering the model’s external interface.

### How does the Hadamard MLP affect training stability?

The orthogonal nature of the Hadamard matrix preserves gradient magnitudes during backpropagation, mitigating vanishing or exploding gradients common in very deep networks. Additionally, the severe parameter reduction (from `n²` to `3n`) acts as a strong structural regularizer, often improving generalization on smaller datasets compared to over‑parameterized dense MLPs.

### Can Hadamard MLPs be quantized for inference?

Yes. The [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) file (lines 58‑62) demonstrates how the Hadamard transform is preserved during quantization export. Because the transform matrix **H** is fixed and orthogonal, it does not require quantization itself—only the three diagonal vectors (`d1`, `d2`, `d3`) need to be quantized, further reducing the memory footprint for deployment.

### Is the Hadamard MLP slower than a standard dense layer on GPUs?

While the theoretical complexity is lower (`O(n log n)` vs `O(n²)`), the actual speedup depends on the hardware’s implementation of the fast Walsh‑Hadamard transform. On modern accelerators with optimized FWHT kernels, the Hadamard MLP typically demonstrates lower memory bandwidth usage and faster inference for moderate hidden sizes (512–2048 dimensions), though dense matrix multiplications may remain faster on specific tensor cores optimized for standard GEMM operations.