Hadamard MLP vs Standard FFN in Needle 2: Architectural and Computational Differences

Needle 2 replaces the conventional two-layer feed-forward network with a Hadamard MLP that uses fixed Walsh-Hadamard transforms and learnable diagonal scalings, reducing complexity from O(d_model·d_ff) to O(d_model·log d_model) while maintaining expressive power.

The cactus-compute/needle repository implements a novel transformer architecture in Needle 2 that substitutes the traditional feed-forward network (FFN) with a more efficient Hadamard MLP. This architectural shift fundamentally changes how hidden representations are transformed between attention layers, trading dense matrix multiplications for structured orthogonal transforms. The implementation in needle/model/architecture.py demonstrates how lightweight deployment targets can achieve significant efficiency gains without sacrificing model capacity.

Core Architectural Differences

Layer Composition and Transformations

Standard transformer FFN blocks follow a predictable pattern: two dense projections separated by a non-linearity. In contrast, the Hadamard MLP decomposes the transformation into a sequence of diagonal scalings and fixed orthogonal transforms.

Standard FFN structure:

• Linear projection up (Dense with expansion factor, typically 4×)
• Non-linear activation (ReLU or SiLU)
• Linear projection down (back to d_model)

Hadamard MLP structure:

• Diagonal scaling (d1) followed by Walsh-Hadamard transform (H)
• Second diagonal scaling (d2) followed by SiLU activation
• Second Walsh-Hadamard transform (H) followed by final diagonal scaling (d3)

This restructuring means the Hadamard MLP performs two passes through the Walsh-Hadamard matrix with learned scaling operations between them, rather than learning full weight matrices.

Weight Structure and Parameter Efficiency

The most significant architectural difference lies in the parameterization strategy. According to the source code in needle/model/architecture.py, the Hadamard MLP learns only three diagonal vectors while utilizing a fixed, deterministic transform matrix.

Standard FFN parameters:

• Two full dense matrices (W₁, W₂) with shape [d_model, d_ff] and [d_ff, d_model]
• Parameter count scales with d_model × d_ff (typically 4 × d_model²)

Hadamard MLP parameters:

• Three diagonal vectors (d1, d2, d3) of length n (power-of-2 padded d_model)
• One fixed Walsh-Hadamard matrix H generated on-the-fly by _walsh_matrix()
• Total learnable parameters: 3 × n (where n ≈ d_model)

The matrix H is orthogonal and requires no storage or gradient computation, instantiated via the helper function defined in the same file.

Computational Complexity and Memory Usage

Complexity Analysis

The Hadamard MLP fundamentally changes the computational bottleneck of the transformer block. While standard FFNs are memory-bandwidth bound due to large matrix multiplications, the Hadamard version leverages fast transform algorithms.

Standard FFN complexity:

• Two matrix multiplications: O(d_model · d_ff) each
• With typical expansion factor of 4: O(4 · d_model²)

Hadamard MLP complexity:

• Fast Walsh-Hadamard Transform: O(n · log n) where n ≈ d_model
• Three diagonal multiplications: O(n) each
• Total: O(d_model · log d_model), a significant reduction for large models

Memory Footprint

Storage requirements drop proportionally with the parameter reduction. Standard FFN implementations must maintain two large weight tensors in memory during training and inference. The Hadamard MLP stores only three parameter vectors of size approximately d_model, dramatically reducing memory pressure for edge deployments.

Implementation in Needle 2

The Hadamard MLP integrates directly into the transformer block structure in needle/model/architecture.py, replacing the traditional FFN sub-module after the attention mechanism.

Block Integration

The Block class inserts the Hadamard MLP after attention and normalization:

# needle/model/architecture.py - Block.__call__

class Block(nn.Module):
    ...
    @nn.compact
    def __call__(self, x, mask=None, rope=None, quant=False, engram_kv=None, site_flags=None):
        # Attention mechanism omitted for brevity

        skip = x
        x = ZCRMSNorm(dtype=self.dtype, name="pre_hada_norm")(x)
        # Hadamard MLP replaces traditional FFN

        x = HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)
        return skip + x

HadamardMLP Implementation

The core implementation (lines 87-103) constructs the transform using JAX operations:

# needle/model/architecture.py - HadamardMLP class

class HadamardMLP(nn.Module):
    d_model: int
    dtype: jnp.dtype = jnp.bfloat16

    @nn.compact
    def __call__(self, x):
        n = 1 << (self.d_model - 1).bit_length()          # Power-of-2 size

        H = _walsh_matrix(n).astype(self.dtype)          # Fixed Hadamard matrix

        d1 = self.param("d1", jinit.ones, (n,)).astype(self.dtype)
        d2 = self.param("d2", jinit.ones, (n,)).astype(self.dtype)
        d3 = self.param("d3", jinit.constant(0.02), (n,)).astype(self.dtype)

        # Pad to power of 2 if necessary

        pad = n - self.d_model
        z = jnp.pad(x, ((0, 0), (0, 0), (0, pad))) if pad else x
        
        # Core Hadamard transforms with diagonal scaling

        z = (d1 * z) @ H                     # First scaling + Hadamard

        z = nn.silu(d2 * z) @ H              # SiLU + second scaling + Hadamard

        return (d3 * z)[..., : self.d_model] # Final scaling and crop

Contrast with Standard FFN

For reference, a conventional implementation would follow this pattern:

class StandardFFN(nn.Module):
    d_model: int
    d_ff: int = None  # Typically 4 * d_model

    dtype: jnp.dtype = jnp.bfloat16

    @nn.compact
    def __call__(self, x):
        if self.d_ff is None:
            self.d_ff = 4 * self.d_model
            
        x = nn.Dense(self.d_ff, dtype=self.dtype)(x)      # Up-projection

        x = nn.silu(x)                                     # Activation

        x = nn.Dense(self.d_model, dtype=self.dtype)(x)   # Down-projection

        return x

Summary

• The Hadamard MLP replaces dense matrix multiplications with fixed Walsh-Hadamard transforms and learnable diagonal scalings, reducing parameters from O(d_model²) to O(d_model).
• Computational complexity drops from O(d_model · d_ff) to O(d_model · log d_model) due to the fast transform algorithm, making inference significantly faster.
• Implementation resides in needle/model/architecture.py where the HadamardMLP class integrates seamlessly into the transformer Block, substituting for the traditional two-layer FFN.
• Memory footprint is minimized by storing only three parameter vectors rather than two full weight matrices, crucial for the lightweight deployment targets of Needle 2.
• Non-linearity placement follows standard practice (SiLU activation) but operates between two orthogonal transforms, preserving signal norm while maintaining expressiveness.

Frequently Asked Questions

What is the primary advantage of Hadamard MLP over standard FFN?

The primary advantage is computational and memory efficiency. While standard FFNs require storing and multiplying two dense matrices of size [d_model, 4*d_model], the Hadamard MLP uses fixed orthogonal transforms and learns only three diagonal vectors. This reduces both memory bandwidth pressure and floating-point operations during inference, making it ideal for resource-constrained environments without significantly impacting model capacity.

How does the Walsh-Hadamard transform reduce computational cost?

The Walsh-Hadamard transform can be computed using a Fast Walsh-Hadamard Transform (FWHT) algorithm similar to the FFT, operating in O(n log n) time rather than the O(n²) required by matrix multiplication. In needle/model/architecture.py, the _walsh_matrix() function generates the transform matrix, but the actual implementation leverages efficient matrix multiplication primitives that exploit the recursive structure of Hadamard matrices, avoiding explicit storage of the full n × n matrix.

Does the Hadamard MLP maintain the same model capacity as a standard FFN?

Yes, the architecture maintains comparable expressive power despite the parameter reduction. The combination of learnable diagonal scalings (d1, d2, d3) and fixed orthogonal transforms preserves the ability to model complex non-linear relationships. The orthogonal nature of the Hadamard matrix ensures signal norms are preserved, while the three stages of scaling provide sufficient degrees of freedom to approximate the behavior of dense projections in standard FFNs.

Why does Needle 2 pad the input to power-of-2 dimensions?

The Hadamard MLP requires power-of-2 dimensions because Walsh-Hadamard matrices are naturally defined for sizes 2^k. In the HadamardMLP.__call__ method, the code calculates n = 1 << (self.d_model - 1).bit_length() to find the next power of two, then pads the input if necessary. This padding ensures compatibility with the fast transform implementation while the final slicing operation [..., : self.d_model] removes the padding before returning the result, maintaining the original model dimensionality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}