How the Hadamard MLP Differs from Standard Feed-Forward Networks in Needle 2

The Hadamard MLP replaces dense weight matrices with Walsh-Hadamard transforms and learnable diagonal scalings, reducing parameters from O(d²) to O(d) while maintaining expressivity through structured linear algebra.

The Needle 2 framework implements a novel feed-forward architecture that diverges from conventional transformer designs. Instead of standard two-layer dense networks, it employs a Hadamard MLP that leverages fast Walsh-Hadamard transforms to achieve significant parameter efficiency and computational speedups.

Architectural Comparison: Hadamard MLP vs. Standard FFN

Weight Representation and Parameter Efficiency

Standard feed-forward networks in transformers rely on two learned dense matrices (nn.Dense) requiring O(d_model²) parameters—specifically approximately 2·d_model² parameters for the two linear layers. In contrast, the Hadamard MLP in Needle 2 constructs its weight matrices implicitly through structured decomposition.

According to the source code in needle/model/architecture.py (lines 288-302), the Hadamard MLP represents weights as:

W = diag(d₁) · H · diag(d₂) · H · diag(d₃)

Where H is a fixed Walsh-Hadamard matrix and only the three diagonal vectors (d₁, d₂, d₃) are learned. This reduces the parameter count to approximately 3·n (where n is the padded dimension)—an order-of-magnitude reduction for typical model sizes.

Computational Complexity

Standard FFNs perform matrix multiplication with O(d²) complexity. The Hadamard MLP replaces dense multiplications with fast Hadamard transforms that operate in O(n log n) time using only element-wise operations and the fixed orthogonal matrix.

The implementation handles dimension alignment by padding inputs to the next power-of-two (n = 1 << (d_model-1).bit_length()), ensuring the Hadamard matrix remains square. The padding is stripped after the final transform, maintaining the original model dimension throughout the block.

Activation Functions and Non-Linearity

While standard feed-forward networks typically place GELU or ReLU activations between two dense layers, the Hadamard MLP integrates SiLU activation differently. As implemented in the HadamardMLP class, the non-linearity is applied after the first Hadamard transform:

nn.silu(d2 * z) @ H

This placement ensures non-linear transformation occurs within the structured Hadamard computation flow rather than between separate dense projections.

Implementation in the Needle 2 Codebase

The HadamardMLP class is defined in needle/model/architecture.py (lines 288-302) and integrated into the transformer Block at line 336. Unlike standard implementations that instantiate nn.Dense layers directly within blocks, Needle 2 invokes the Hadamard MLP as a standalone module:

# Integration inside a Needle transformer block

from needle.model.architecture import HadamardMLP
import jax.numpy as jnp
import flax.linen as nn

class TransformerBlock(nn.Module):
    d_model: int = 512
    dtype: jnp.dtype = jnp.bfloat16

    @nn.compact
    def __call__(self, x):
        # Standard attention would go here

        # ...

        
        # Hadamard-based feed-forward network

        x = HadamardMLP(
            self.d_model, 
            self.dtype, 
            name="hadamard_mlp"
        )(x)
        return x

The underlying _walsh_matrix helper generates the fixed orthogonal matrix used in the transforms. For direct usage of the low-level primitives:

from needle.model.architecture import _walsh_matrix
import jax.numpy as jnp

def fast_hadamard_transform(x):
    """Apply Walsh-Hadamard transform with power-of-2 padding."""
    n = 1 << (x.shape[-1] - 1).bit_length()
    H = _walsh_matrix(n).astype(x.dtype)
    
    # Pad to next power of two if needed

    pad = n - x.shape[-1]
    if pad:
        x = jnp.pad(x, ((0, 0), (0, pad)))
    
    return x @ H

# Example usage

vec = jnp.arange(8, dtype=jnp.float32)[None, :]
transformed = fast_hadamard_transform(vec)

Performance Characteristics

The Hadamard MLP offers distinct computational advantages:

• Memory footprint: Stores only three diagonal vectors instead of two full weight matrices
• Cache efficiency: Walsh-Hadamard transforms consist of simple element-wise operations that utilize cache lines effectively
• Orthogonal properties: The Hadamard matrix H is orthogonal by construction, providing stable gradient flow without additional normalization techniques

As noted in the codebase context within needle/model/export.py, this architecture also facilitates specific quantization strategies due to its structured weight representation.

Summary

• Parameter reduction: The Hadamard MLP reduces feed-forward parameters from 2·d_model² to 3·n using diagonal scaling and fixed Walsh-Hadamard matrices.
• Computational efficiency: Replaces O(d²) dense matrix multiplication with O(n log n) fast transforms.
• Implementation location: Defined in needle/model/architecture.py (lines 288-302) and integrated into transformer blocks at line 336.
• Dimension handling: Automatically pads inputs to power-of-two dimensions to accommodate square Hadamard matrices.
• Activation strategy: Employs SiLU activation within the Hadamard computation flow rather than between separate layers.

Frequently Asked Questions

Why does the Hadamard MLP require power-of-two dimensions?

The Walsh-Hadamard matrix must be square and orthogonal, requiring dimensions that are powers of two (2^k). The implementation in needle/model/architecture.py automatically pads inputs to n = 1 << (d_model-1).bit_length() and strips the padding after computation, ensuring compatibility with arbitrary model dimensions while maintaining the mathematical properties of the transform.

How does the Hadamard MLP maintain expressivity with fewer parameters?

While standard dense layers learn arbitrary linear transformations, the Hadamard MLP learns specific diagonal scaling vectors that modulate fixed orthogonal transforms. The composition diag(d₁)·H·diag(d₂)·H·diag(d₃) provides sufficient expressive power for transformer feed-forward layers because the diagonal parameters can rescale and gate the frequency components introduced by the Walsh-Hadamard basis.

Can the Hadamard MLP replace standard FFNs in existing transformer architectures?

Yes, the HadamardMLP class in Needle 2 implements the standard Flax linen module interface, making it a drop-in replacement for nn.Dense layers in feed-forward networks. It accepts d_model and dtype parameters and returns tensors of the same shape as the input, maintaining API compatibility with conventional transformer blocks.

What hardware considerations apply to the Hadamard MLP?

The Hadamard transform consists entirely of addition and subtraction operations (no multiplications in the transform itself), making it exceptionally efficient on hardware with limited multiply-accumulate units. The element-wise nature of the fast Walsh-Hadamard transform also enables vectorization across modern accelerators, though the power-of-two padding requirement may require additional memory allocation during forward passes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}