How the Hadamard MLP Is Implemented in Needle 2: Architecture and Code Walkthrough

Needle 2 replaces the traditional feed-forward network with a HadamardMLP that leverages the Walsh-Hadamard transform and learnable diagonal scalings to perform fast, orthogonal feature mixing without dense matrix multiplications.

The HadamardMLP is a core innovation in the Needle 2 transformer architecture from the cactus-compute/needle repository. Unlike standard MLPs that rely on expensive dense linear projections, the Hadamard MLP implementation in Needle 2 uses structured matrix operations to reduce computational overhead while maintaining expressive power. This article examines the source code in needle/model/architecture.py to explain exactly how this efficient feed-forward layer works.

Core Architecture of HadamardMLP

The HadamardMLP class defined in needle/model/architecture.py (lines 87-103) reimagines the feed-forward block as a sequence of diagonal scalings interleaved with Walsh-Hadamard transforms.

Power-of-Two Padding and Walsh-Hadamard Matrix Generation

The implementation first rounds the input dimension d_model up to the nearest power of two, denoted as n. It then generates a Walsh-Hadamard matrix H of size n × n via the internal _walsh_matrix function, caching this as a constant tensor converted to the appropriate dtype (H = _walsh_matrix(n).astype(self.dtype)). This orthogonal matrix enables the O(n log n) fast Walsh-Hadamard transform instead of the O(n²) cost of dense matrix multiplication.

Learnable Diagonal Scaling Vectors

Three trainable parameters—d1, d2, and d3—are initialized as diagonal vectors of length n. According to the source code, d1 and d2 use jax.nn.initializers.ones, while d3 initializes to a constant value of 0.02. These vectors act as per-feature gain and bias terms applied before and after the transform operations, allowing the model to learn feature-wise scaling while preserving the orthogonal structure of the Hadamard transform.

Two-Stage Hadamard Mixing Forward Pass

The forward pass executes a three-step mixing process:

1. First Transform: The padded input is element-wise multiplied by d1, then matrix-multiplied with the Walsh-Hadamard matrix H (z = (d1 * padded_x) @ H).
2. Non-linearity and Second Transform: The result is scaled by d2, passed through a SiLU activation, and transformed again by H (z = nn.silu(d2 * z) @ H).
3. Final Scaling: The output is scaled by d3 and truncated back to the original d_model dimension (output = (d3 * z)[..., : d_model]).

Integration into the Transformer Block

Inside the Block class in needle/model/architecture.py, the HadamardMLP is invoked immediately after the self-attention mechanism. The code passes the token tensor through HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x) and adds the result to the residual stream via skip + x. This placement follows the standard transformer pattern but substitutes the Hadamard-based feed-forward path for the traditional dense MLP.

Implementation Details in needle/model/architecture.py

The complete implementation spans lines 87-103 of needle/model/architecture.py, where the HadamardMLP class encapsulates the logic. The _walsh_matrix helper function generates the binary-valued orthogonal matrix required for the transform. The Block class (lines 34-38) demonstrates the integration point, while SimpleAttentionNetwork (lines 78-86) constructs the full model stack that utilizes this architecture.

Practical Usage Examples

Standalone HadamardMLP

import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# Dummy batch: (batch, seq_len, d_model)

x = jnp.ones((2, 16, 512), dtype=jnp.bfloat16)

# Initialise the module

hadamard = HadamardMLP(d_model=512, dtype=jnp.bfloat16)

# Initialise parameters with a random key

variables = hadamard.init(jax.random.PRNGKey(0), x)

# Apply the Hadamard‑MLP

y = hadamard.apply(variables, x)
print(y.shape)          # → (2, 16, 512)

Full Transformer with HadamardMLP

from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

cfg = TransformerConfig(d_model=512, num_heads=8, num_layers=12)
net = SimpleAttentionNetwork(config=cfg)

# Dummy token IDs

tokens = jnp.arange(0, 64).reshape(1, 64)  # shape (batch, seq_len)

# Initialise & run

params = net.init(jax.random.PRNGKey(0), tokens)
logits = net.apply(params, tokens)  # logits = attention + HadamardMLP output

Summary

• The HadamardMLP replaces dense weight matrices with the Walsh-Hadamard transform, reducing computational complexity from O(n²) to O(n log n) for the mixing operations.
• Three learnable diagonal vectors (d1, d2, d3) provide trainable per-feature scaling while maintaining orthogonal mixing properties.
• The implementation resides in needle/model/architecture.py (lines 87-103) and integrates into the transformer via the Block class after the attention mechanism.
• The layer automatically handles power-of-two padding and truncation to accommodate arbitrary d_model dimensions.

Frequently Asked Questions

What is a Hadamard MLP and why does Needle 2 use it?

A Hadamard MLP is a feed-forward neural network layer that uses the Walsh-Hadamard transform instead of dense matrix multiplication to mix features across the hidden dimension. Needle 2 uses this architecture to reduce computational cost while maintaining the expressiveness of traditional MLPs through learnable diagonal scalings.

How does the Walsh-Hadamard transform improve efficiency over standard MLPs?

The Walsh-Hadamard transform can be computed in O(n log n) time using fast matrix operations, whereas standard MLP dense layers require O(n²) operations for matrix multiplication. This reduces the computational bottleneck in the feed-forward layers while preserving the orthogonal properties needed for stable gradient flow.

Where is the HadamardMLP class defined in the Needle repository?

The HadamardMLP class is defined in needle/model/architecture.py between lines 87 and 103. This file also contains the _walsh_matrix helper function and the Block class that integrates HadamardMLP into the complete transformer architecture.

Can I use HadamardMLP as a standalone module outside the Transformer?

Yes, the HadamardMLP class is a standard Flax module that can be instantiated independently with HadamardMLP(d_model=512, dtype=jnp.bfloat16). It accepts tensors of shape (batch, seq_len, d_model) and can be used in any JAX-based architecture requiring efficient feature mixing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}