What Is a Hadamard MLP and How Does It Differ From Traditional Feed‑Forward Networks?

A Hadamard MLP is a parameter‑efficient feed‑forward layer that replaces dense weight matrices with three learnable diagonal scaling vectors and the fixed Walsh‑Hadamard transform, reducing trainable parameters from O(n²) to O(n) while maintaining expressiveness through structured orthogonal mixing.

The Hadamard MLP is a specialized component in the Needle transformer model developed by cactus-compute/needle. Unlike conventional feed‑forward networks that rely on arbitrary dense linear projections, this layer exploits the orthogonal structure of the Walsh‑Hadamard matrix to achieve dramatic memory and compute savings. Understanding its implementation reveals how structured sparsity can replace brute‑force parameter counts in modern transformer architectures.

Core Architecture and Design

The fundamental innovation of the Hadamard MLP lies in its decomposition of the weight matrix. Rather than learning a full dense matrix W of size d_model × d_model, the layer learns only three diagonal scaling vectors—d1, d2, and d3—and applies the fixed Walsh‑Hadamard transform (H) repeatedly to mix information across the hidden dimension.

The transform matrix H is orthogonal and defined recursively, allowing multiplication in O(n log n) time via the fast Walsh‑Hadamard transform (FWHT). In needle/model/architecture.py, the helper function _walsh_matrix generates this normalized matrix by padding the input dimension to the next power of two (n), ensuring efficient transform operations on hardware accelerators.

Implementation Details in Needle

According to the source code in needle/model/architecture.py (lines 87‑103), the HadamardMLP class implements the following data flow:

1. Padding: The input dimension (d_model) is padded to the next power‑of‑two n to accommodate the Walsh‑Hadamard matrix requirements.
2. First mixing: The input is scaled by d1 and multiplied by the Hadamard matrix H.
3. Non‑linearity: The result is scaled by d2 and passed through the SiLU activation (nn.silu).
4. Second mixing: Another Hadamard transform is applied.
5. Output projection: The final vector d3 scales the output, which is then truncated back to the original d_model dimension.

# Conceptual data flow inside HadamardMLP.__call__

z = (d1 * padded_x) @ H            # Linear mixing via Hadamard

z = nn.silu(d2 * z) @ H            # Non-linearity + second mixing

out = (d3 * z)[..., :d_model]      # Final scaling and truncation

The three diagonal vectors are defined as learnable parameters using Flax’s self.param() convention, appearing as ('d1',), ('d2',), and ('d3',) in the parameter tree. This contrasts sharply with standard dense layers that instantiate full weight matrices and bias vectors.

Comparison: Hadamard MLP vs. Traditional Feed‑Forward Networks

Traditional MLP layers compute y = activation(W·x + b), where W is a dense matrix requiring O(d_model²) parameters and O(d_model²) floating‑point operations per token. The Hadamard MLP introduces three critical differences:

• Parameter Efficiency: Only 3 × n scalars are learned versus n² dense weights. For a typical d_model of 768 (padded to n = 1024), this reduces parameters from ~590K to ~3K in the feed‑forward component.
• Computational Complexity: Multiplication by H uses the fast Walsh‑Hadamard transform, requiring O(n log n) operations instead of the O(n²) matrix‑vector product.
• Structural Constraint: The effective weight matrix is implicitly D3 · H · D2 · H · D1, where D represents diagonal matrices. This enforces a specific low‑rank structure that acts as a strong regularizer, preventing overfitting compared to arbitrary dense configurations.

Practical Usage Examples

Instantiating a Stand‑alone Hadamard MLP

For debugging or experimental architectures, you can import and call the layer directly from needle.model.architecture:

import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# d_model = 768 (typical for small transformers)

mlp = HadamardMLP(d_model=768, dtype=jnp.bfloat16)

# Dummy input: (batch, seq_len, d_model)

x = jnp.ones((2, 10, 768), dtype=jnp.bfloat16)
y = mlp(x)          # → (2, 10, 768)

Integration Within a Full Transformer Stack

In production usage, the HadamardMLP is automatically invoked within the Block class (lines 34‑38 of architecture.py) when constructing the model:

from needle.model.architecture import Stack, TransformerConfig

cfg = TransformerConfig(
    num_layers=12,
    d_model=768,
    num_heads=12,
    num_kv_heads=12,
)

# Build full model; HadamardMLP is used inside each Block

model = Stack(config=cfg)

inputs = jnp.ones((1, 20, cfg.d_model), dtype=jnp.bfloat16)
logits, _ = model(inputs)   # logits shape: (1, 20, cfg.d_model)

Manual Replacement of Standard FFN

To substitute a traditional feed‑forward network with the Hadamard variant in a custom Block subclass:

from needle.model.architecture import Block

class SimpleBlock(Block):
    @nn.compact
    def __call__(self, x, **kw):
        # Run attention from parent class

        x = super().__call__(x, **kw)
        # Replace standard MLP with Hadamard version

        x = HadamardMLP(self.d_model, self.dtype)(x)
        return x

Summary

• The Hadamard MLP reduces feed‑forward parameters from quadratic to linear complexity by learning only three diagonal vectors (d1, d2, d3) instead of dense matrices.
• It utilizes the Walsh‑Hadamard transform (_walsh_matrix in needle/model/architecture.py) to achieve O(n log n) mixing operations via the fast transform algorithm.
• The layer is implemented in Flax and integrated into the Needle transformer stack at Block level (lines 34‑38).
• Trade‑offs: Significant memory savings and regularization benefits versus the expressiveness of unconstrained dense projections.

Frequently Asked Questions

Why pad to the next power of two?

The Walsh‑Hadamard transform requires matrix dimensions that are powers of two to maintain its recursive, balanced structure. In needle/model/architecture.py, the implementation pads d_model to n (the next power of two) and truncates the output back to the original dimension after the final scaling step, ensuring computational efficiency without altering the model’s external interface.

How does the Hadamard MLP affect training stability?

The orthogonal nature of the Hadamard matrix preserves gradient magnitudes during backpropagation, mitigating vanishing or exploding gradients common in very deep networks. Additionally, the severe parameter reduction (from n² to 3n) acts as a strong structural regularizer, often improving generalization on smaller datasets compared to over‑parameterized dense MLPs.

Can Hadamard MLPs be quantized for inference?

Yes. The needle/model/export.py file (lines 58‑62) demonstrates how the Hadamard transform is preserved during quantization export. Because the transform matrix H is fixed and orthogonal, it does not require quantization itself—only the three diagonal vectors (d1, d2, d3) need to be quantized, further reducing the memory footprint for deployment.

Is the Hadamard MLP slower than a standard dense layer on GPUs?

While the theoretical complexity is lower (O(n log n) vs O(n²)), the actual speedup depends on the hardware’s implementation of the fast Walsh‑Hadamard transform. On modern accelerators with optimized FWHT kernels, the Hadamard MLP typically demonstrates lower memory bandwidth usage and faster inference for moderate hidden sizes (512–2048 dimensions), though dense matrix multiplications may remain faster on specific tensor cores optimized for standard GEMM operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}