Hadamard MLP vs Standard FFN: Architecture and Implementation in Needle

A Hadamard MLP replaces the dense learnable weight matrix of a standard feed-forward network with a fixed Walsh-Hadamard matrix paired with learnable diagonal scaling factors, reducing trainable parameters from quadratic to linear complexity while maintaining representational capacity through orthogonal transformations.

The cactus-compute/needle repository implements a parameter-efficient feed-forward variant called Hadamard MLP to optimize transformer memory footprint and inference speed. Unlike conventional feed-forward networks (FFNs) that rely on dense matrix multiplications, this approach leverages the fast Hadamard transform to achieve sub-quadratic computational complexity. The implementation resides primarily in needle/model/architecture.py, where the HadamardMLP class serves as a drop-in replacement for standard FFN layers.

What Is a Hadamard MLP?

A Hadamard MLP is a neural network layer that performs linear transformations using a fixed orthogonal matrix rather than a learned dense weight matrix. Mathematically, while a standard FFN computes y = x @ W + b using a learned dense matrix W, the Hadamard variant computes:

y = (D1 * x) @ H @ D2 + b

Where:

• H is a fixed, normalized Walsh-Hadamard matrix (orthogonal, entries ±1)
• D1 and D2 are learnable diagonal scaling matrices
• b is the bias vector

Because H is orthogonal and constant, the layer preserves input norms while the learnable diagonals D1 and D2 provide sufficient flexibility to adapt the representation. This structure appears in needle/model/architecture.py at line 287, where the HadamardMLP class encapsulates this logic.

Hadamard MLP vs Standard FFN: Key Differences

Characteristic	Standard FFN	Hadamard MLP
Weight Representation	Dense matrix W (learned)	Fixed Hadamard matrix H with learnable diagonal scalings D1, D2
Parameter Count	O(N²) quadratic in hidden dimension	O(N) linear (only diagonal values stored)
Computational Complexity	O(N²) matrix multiplication	O(N log N) via fast Hadamard transform
Memory Footprint	Stores full N×N weight tensors	Stores two diagonal vectors (2N parameters)
Orthogonality	Not guaranteed	Preserved by construction via H

The orthogonal property of the Hadamard matrix ensures that activations maintain their norm distribution through the layer, which aids training stability. According to the Needle source code, this design choice enables aggressive quantization strategies, as noted in needle/model/export.py regarding Hadamard diagonals.

Implementation Details in Needle

In the Needle codebase, the HadamardMLP class is defined at line 287 in needle/model/architecture.py. This implementation provides the same interface as standard PyTorch nn.Linear layers, allowing seamless substitution within transformer blocks.

The layer is integrated into the model architecture at line 336 of the same file, where it replaces conventional feed-forward sublayers. The repository also references Hadamard-specific optimizations in needle/model/export.py (around line 6), which documents how the diagonal structure facilitates weight quantization and export procedures.

Practical Usage Example

You can instantiate and use a HadamardMLP exactly like a standard FFN layer:

import torch
from needle.model.architecture import HadamardMLP

# Create dummy input (batch, sequence, features)

x = torch.randn(2, 16, 512)

# Initialize Hadamard MLP with equivalent dimensions

hadamard_mlp = HadamardMLP(d_model=512, dtype=torch.float16, name="hadamard_mlp")

# Forward pass computes (D1 * x) @ H @ D2 + b

y = hadamard_mlp(x)

print(y.shape)  # torch.Size([2, 16, 512])

Swapping a standard FFN for a HadamardMLP requires only changing the class instantiation; the forward pass API remains identical, making it compatible with existing transformer implementations.

Summary

• Parameter Efficiency: Hadamard MLPs reduce trainable parameters from O(N²) to O(N) by learning only diagonal scalings rather than full dense matrices.
• Computational Speed: The fast Hadamard transform achieves O(N log N) complexity compared to O(N²) for standard matrix multiplication.
• Fixed Orthogonal Base: The Walsh-Hadamard matrix provides a theoretically grounded, norm-preserving transformation that requires no learning.
• Drop-in Compatibility: In needle/model/architecture.py, the HadamardMLP class implements the same interface as standard feed-forward layers for easy integration.

Frequently Asked Questions

How does a Hadamard MLP reduce memory usage compared to a standard FFN?

A standard FFN must store a dense weight matrix of size N×N, requiring O(N²) memory. The Hadamard MLP stores only two diagonal vectors (D1 and D2) of size N each, reducing storage to O(2N). As implemented in needle/model/architecture.py, this compression enables larger model capacities within the same memory constraints.

Can I replace any standard FFN with a Hadamard MLP in existing models?

Yes, the HadamardMLP class in Needle is designed as a drop-in replacement. It accepts the same d_model dimensions and tensor shapes as conventional layers. However, you must ensure that subsequent layers can adapt to the potentially different activation distributions, though the orthogonal property of the Hadamard transform generally maintains stable norms.

What is the computational complexity of the Hadamard transform?

The fast Walsh-Hadamard transform computes the matrix multiplication with the fixed Hadamard matrix in O(N log N) time, compared to O(N²) for dense matrix multiplication. This logarithmic complexity makes Hadamard MLPs significantly faster for large hidden dimensions, a key optimization target in the Needle repository's architecture.

Where does Needle handle quantization for Hadamard MLP weights?

The needle/model/export.py file references Hadamard diagonals in the context of model quantization and export. Because the Hadamard matrix itself is fixed and requires no storage, quantization efforts focus solely on the diagonal scaling factors (D1 and D2), simplifying the quantization pipeline compared to full matrix quantization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}