Hadamard MLP in Needle 2: Efficient Feed-Forward Networks for Transformers

The Hadamard MLP is a parameter-efficient feed-forward module that replaces traditional dense weight matrices with fixed Walsh-Hadamard transforms and three learnable diagonal scaling vectors, drastically reducing memory usage while maintaining expressive non-linear transformations in transformer blocks.

The Hadamard MLP serves as the core feed-forward component in Needle 2, an open-source transformer implementation by cactus-compute designed for efficient inference on resource-constrained hardware. Unlike conventional MLPs that learn full weight matrices, this architecture leverages orthogonal Hadamard transforms to minimize trainable parameters while preserving model capacity. According to the source code in needle/model/architecture.py, the implementation combines fixed mathematical transforms with strategic parameterization to achieve computational efficiency.

What is a Hadamard MLP?

A Hadamard MLP is a specialized neural network module that performs dense linear transformations without learning full weight matrices. Instead, it utilizes a fixed Walsh-Hadamard matrix—an orthogonal matrix that enables fast, structured transforms—combined with learned diagonal scaling vectors to modulate signal flow.

The key innovation lies in decoupling the transformation structure (fixed Hadamard matrix) from the learnable capacity (diagonal scales). This approach reduces the parameter count from quadratic (O(d^2)) in standard dense layers to linear (O(d)), making it ideal for deployment scenarios where memory bandwidth and storage are limiting factors.

Implementation in Needle 2

Core Architecture in architecture.py

The HadamardMLP class is defined in needle/model/architecture.py at lines 87-103 as a Flax module inheriting from nn.Module. The constructor builds a Walsh-Hadamard matrix (_walsh_matrix) sized to the next power-of-two greater than the model dimension, ensuring the transform remains orthogonal and computationally efficient.

The module defines three learned diagonal scaling vectors:

• d1: Pre-scale vector applied before the first Hadamard transform
• d2: Non-linearity scale vector applied within the activation function
• d3: Post-scale vector applied after the second Hadamard transform

Forward Pass Mechanics

The forward implementation follows a strict five-step pipeline that maintains the input dimension throughout while enabling rich non-linear interactions:

1. Pre-scaling: The input x is element-wise multiplied by d1 (d1 * x)
2. First Hadamard: The scaled input is multiplied by the fixed Walsh-Hadamard matrix
3. Activated scaling: The result is multiplied by d2 and passed through nn.silu activation (nn.silu(d2 * ...))
4. Second Hadamard: Another multiplication by the Hadamard matrix
5. Post-scaling and truncation: The output is multiplied by d3 and truncated to the original model dimension

Because the Hadamard matrix remains fixed and orthogonal, the only trainable parameters are the three diagonal vectors, reducing total parameters by orders of magnitude compared to standard feed-forward networks.

Integration into Transformer Blocks

Within each Block (Needle 2's basic transformer layer), the Hadamard MLP follows the self-attention mechanism. As shown in Block.__call__ at lines 34-37 of needle/model/architecture.py, the integration follows this pattern:

x = ZCRMSNorm(..., name="pre_hada_norm")(x)
x = HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)

The output incorporates a residual connection (skip + x), where the transformed output adds back to the residual stream. This placement gives each layer a computationally efficient feed-forward path that captures non-linear transformations without the memory overhead of conventional dense MLPs. This design choice specifically targets efficient inference on limited-resource hardware, as the fixed Hadamard transform eliminates the need to store large weight matrices during forward passes.

Usage Examples

Direct Instantiation

You can instantiate the Hadamard MLP directly for custom architectures or experimentation:

import jax.numpy as jnp
from needle.model.architecture import HadamardMLP

# Example: a model with hidden size 512

mlp = HadamardMLP(d_model=512, dtype=jnp.bfloat16)

# Dummy input: (batch, seq_len, d_model)

x = jnp.ones((2, 128, 512), dtype=jnp.bfloat16)

# Apply the MLP (inside a Flax module, you would call it in a @nn.compact method)

y = mlp(x)          # y has shape (2, 128, 512)

Within the Full Needle 2 Model

When using the complete Needle 2 transformer, the Hadamard MLP is automatically included in each block:

from needle.cli import main as needle_main

# Load a pre-trained Needle 2 checkpoint (path is illustrative)

model, params = needle_main.load_model("models/needle2.pt")

# Run inference – the internal Block will call HadamardMLP as part of its forward pass

output = model.apply(params, input_tokens)

The needle/model/export.py file demonstrates how the diagonal scaling vectors (d1, d2, d3) are serialized alongside other model weights for deployment, while tests/test_inference.py provides end-to-end validation that the Hadamard transforms function correctly within the full forward pass.

Summary

• The Hadamard MLP replaces dense weight matrices with fixed Walsh-Hadamard transforms in Needle 2's transformer blocks, reducing parameters from quadratic to linear complexity.
• Three diagonal vectors (d1, d2, d3) provide the only learnable parameters, applied before, during (via SiLU activation), and after the Hadamard transforms.
• Implementation resides in needle/model/architecture.py at lines 87-103, with integration into transformer blocks at lines 34-37.
• Memory efficiency makes this architecture suitable for resource-constrained inference, as the orthogonal Hadamard matrix requires no storage for gradients or optimizer states.

Frequently Asked Questions

How does the Hadamard MLP reduce memory usage compared to standard MLPs?

Standard MLPs learn full weight matrices of size (d \times d), requiring (O(d^2)) parameters and optimizer state storage. The Hadamard MLP uses a fixed, orthogonal Walsh-Hadamard matrix that requires no gradient storage, learning only three diagonal vectors of size (d). This reduces trainable parameters from (O(d^2)) to (O(d)), significantly decreasing memory footprint during both training and inference on limited-resource hardware.

Can the Hadamard MLP be used outside of the Needle 2 architecture?

Yes, the HadamardMLP class in needle/model/architecture.py is a standalone Flax module that can be imported and used in any JAX-based neural network. The class accepts standard parameters like d_model and dtype, making it compatible with custom transformer implementations or other architectures requiring efficient feed-forward layers. However, optimal performance requires input dimensions aligned to power-of-two boundaries for efficient Hadamard transform computation.

Why are three separate diagonal scaling vectors necessary?

The three vectors (d1, d2, d3) serve distinct functional roles in the transformation pipeline. d1 modulates the input before the first linear transform, d2 controls the non-linearity through the SiLU activation gate, and d3 scales the output after the second Hadamard transform. This factorization allows the network to learn complex, non-linear mappings despite the fixed nature of the Hadamard matrices, effectively approximating the capacity of dense layers with minimal parameters.

What is the Walsh-Hadamard matrix and why is it fixed?

The Walsh-Hadamard matrix is an orthogonal matrix composed of entries (+1) and (-1) that enables fast, structured linear transforms similar to the Fast Fourier Transform (FFT). It remains fixed (non-trainable) because its orthogonality and structure provide a complete basis for linear transformations; the learnable diagonal scales (d1, d2, d3) adapt this fixed basis to the specific task. This immutability eliminates gradient computation for the matrix itself, further reducing computational overhead during backpropagation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}