# Hadamard MLP vs Standard FFN: Architecture and Implementation in Needle

> Explore Hadamard MLPs and their differences from standard FFNs. Discover how Needle's implementation reduces parameters from quadratic to linear complexity with orthogonal transformations.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-04

---

**A Hadamard MLP replaces the dense learnable weight matrix of a standard feed-forward network with a fixed Walsh-Hadamard matrix paired with learnable diagonal scaling factors, reducing trainable parameters from quadratic to linear complexity while maintaining representational capacity through orthogonal transformations.**

The `cactus-compute/needle` repository implements a parameter-efficient feed-forward variant called **Hadamard MLP** to optimize transformer memory footprint and inference speed. Unlike conventional feed-forward networks (FFNs) that rely on dense matrix multiplications, this approach leverages the fast Hadamard transform to achieve sub-quadratic computational complexity. The implementation resides primarily in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), where the `HadamardMLP` class serves as a drop-in replacement for standard FFN layers.

## What Is a Hadamard MLP?

A **Hadamard MLP** is a neural network layer that performs linear transformations using a fixed orthogonal matrix rather than a learned dense weight matrix. Mathematically, while a standard FFN computes `y = x @ W + b` using a learned dense matrix **W**, the Hadamard variant computes:

```python
y = (D1 * x) @ H @ D2 + b

```

Where:
- **H** is a fixed, normalized Walsh-Hadamard matrix (orthogonal, entries ±1)
- **D1** and **D2** are learnable diagonal scaling matrices
- **b** is the bias vector

Because **H** is orthogonal and constant, the layer preserves input norms while the learnable diagonals **D1** and **D2** provide sufficient flexibility to adapt the representation. This structure appears in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at line 287, where the `HadamardMLP` class encapsulates this logic.

## Hadamard MLP vs Standard FFN: Key Differences

| Characteristic | Standard FFN | Hadamard MLP |
|-----------------|--------------|--------------|
| **Weight Representation** | Dense matrix **W** (learned) | Fixed Hadamard matrix **H** with learnable diagonal scalings **D1**, **D2** |
| **Parameter Count** | O(N²) quadratic in hidden dimension | O(N) linear (only diagonal values stored) |
| **Computational Complexity** | O(N²) matrix multiplication | O(N log N) via fast Hadamard transform |
| **Memory Footprint** | Stores full N×N weight tensors | Stores two diagonal vectors (2N parameters) |
| **Orthogonality** | Not guaranteed | Preserved by construction via **H** |

The orthogonal property of the Hadamard matrix ensures that activations maintain their norm distribution through the layer, which aids training stability. According to the Needle source code, this design choice enables aggressive quantization strategies, as noted in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) regarding Hadamard diagonals.

## Implementation Details in Needle

In the Needle codebase, the `HadamardMLP` class is defined at **line 287** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This implementation provides the same interface as standard PyTorch `nn.Linear` layers, allowing seamless substitution within transformer blocks.

The layer is integrated into the model architecture at **line 336** of the same file, where it replaces conventional feed-forward sublayers. The repository also references Hadamard-specific optimizations in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (around line 6), which documents how the diagonal structure facilitates weight quantization and export procedures.

## Practical Usage Example

You can instantiate and use a `HadamardMLP` exactly like a standard FFN layer:

```python
import torch
from needle.model.architecture import HadamardMLP

# Create dummy input (batch, sequence, features)

x = torch.randn(2, 16, 512)

# Initialize Hadamard MLP with equivalent dimensions

hadamard_mlp = HadamardMLP(d_model=512, dtype=torch.float16, name="hadamard_mlp")

# Forward pass computes (D1 * x) @ H @ D2 + b

y = hadamard_mlp(x)

print(y.shape)  # torch.Size([2, 16, 512])

```

Swapping a standard FFN for a `HadamardMLP` requires only changing the class instantiation; the forward pass API remains identical, making it compatible with existing transformer implementations.

## Summary

- **Parameter Efficiency**: Hadamard MLPs reduce trainable parameters from O(N²) to O(N) by learning only diagonal scalings rather than full dense matrices.
- **Computational Speed**: The fast Hadamard transform achieves O(N log N) complexity compared to O(N²) for standard matrix multiplication.
- **Fixed Orthogonal Base**: The Walsh-Hadamard matrix provides a theoretically grounded, norm-preserving transformation that requires no learning.
- **Drop-in Compatibility**: In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `HadamardMLP` class implements the same interface as standard feed-forward layers for easy integration.

## Frequently Asked Questions

### How does a Hadamard MLP reduce memory usage compared to a standard FFN?

A standard FFN must store a dense weight matrix of size N×N, requiring O(N²) memory. The Hadamard MLP stores only two diagonal vectors (D1 and D2) of size N each, reducing storage to O(2N). As implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this compression enables larger model capacities within the same memory constraints.

### Can I replace any standard FFN with a Hadamard MLP in existing models?

Yes, the `HadamardMLP` class in Needle is designed as a drop-in replacement. It accepts the same `d_model` dimensions and tensor shapes as conventional layers. However, you must ensure that subsequent layers can adapt to the potentially different activation distributions, though the orthogonal property of the Hadamard transform generally maintains stable norms.

### What is the computational complexity of the Hadamard transform?

The fast Walsh-Hadamard transform computes the matrix multiplication with the fixed Hadamard matrix in O(N log N) time, compared to O(N²) for dense matrix multiplication. This logarithmic complexity makes Hadamard MLPs significantly faster for large hidden dimensions, a key optimization target in the Needle repository's architecture.

### Where does Needle handle quantization for Hadamard MLP weights?

The [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) file references Hadamard diagonals in the context of model quantization and export. Because the Hadamard matrix itself is fixed and requires no storage, quantization efforts focus solely on the diagonal scaling factors (D1 and D2), simplifying the quantization pipeline compared to full matrix quantization.