# Hadamard MLP vs Standard FFN in Needle 2: Architectural and Computational Differences

> Explore Hadamard MLP vs standard FFN in Needle 2. Discover how Hadamard MLP cuts complexity from O(d_model d_ff) to O(d_model log d_model) while keeping expressive power. Learn the architectural and computational differences.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-12

---

**Needle 2 replaces the conventional two-layer feed-forward network with a Hadamard MLP that uses fixed Walsh-Hadamard transforms and learnable diagonal scalings, reducing complexity from O(d_model·d_ff) to O(d_model·log d_model) while maintaining expressive power.**

The cactus-compute/needle repository implements a novel transformer architecture in Needle 2 that substitutes the traditional feed-forward network (FFN) with a more efficient Hadamard MLP. This architectural shift fundamentally changes how hidden representations are transformed between attention layers, trading dense matrix multiplications for structured orthogonal transforms. The implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) demonstrates how lightweight deployment targets can achieve significant efficiency gains without sacrificing model capacity.

## Core Architectural Differences

### Layer Composition and Transformations

Standard transformer FFN blocks follow a predictable pattern: two dense projections separated by a non-linearity. In contrast, the Hadamard MLP decomposes the transformation into a sequence of diagonal scalings and fixed orthogonal transforms.

**Standard FFN structure:**
- Linear projection up (`Dense` with expansion factor, typically 4×)
- Non-linear activation (ReLU or SiLU)
- Linear projection down (back to `d_model`)

**Hadamard MLP structure:**
- Diagonal scaling (`d1`) followed by Walsh-Hadamard transform (`H`)
- Second diagonal scaling (`d2`) followed by SiLU activation
- Second Walsh-Hadamard transform (`H`) followed by final diagonal scaling (`d3`)

This restructuring means the Hadamard MLP performs two passes through the Walsh-Hadamard matrix with learned scaling operations between them, rather than learning full weight matrices.

### Weight Structure and Parameter Efficiency

The most significant architectural difference lies in the parameterization strategy. According to the source code in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the Hadamard MLP learns only three diagonal vectors while utilizing a fixed, deterministic transform matrix.

**Standard FFN parameters:**
- Two full dense matrices (`W₁`, `W₂`) with shape `[d_model, d_ff]` and `[d_ff, d_model]`
- Parameter count scales with `d_model × d_ff` (typically `4 × d_model²`)

**Hadamard MLP parameters:**
- Three diagonal vectors (`d1`, `d2`, `d3`) of length `n` (power-of-2 padded `d_model`)
- One fixed Walsh-Hadamard matrix `H` generated on-the-fly by `_walsh_matrix()`
- Total learnable parameters: `3 × n` (where `n ≈ d_model`)

The matrix `H` is orthogonal and requires no storage or gradient computation, instantiated via the helper function defined in the same file.

## Computational Complexity and Memory Usage

### Complexity Analysis

The Hadamard MLP fundamentally changes the computational bottleneck of the transformer block. While standard FFNs are memory-bandwidth bound due to large matrix multiplications, the Hadamard version leverages fast transform algorithms.

**Standard FFN complexity:**
- Two matrix multiplications: `O(d_model · d_ff)` each
- With typical expansion factor of 4: `O(4 · d_model²)`

**Hadamard MLP complexity:**
- Fast Walsh-Hadamard Transform: `O(n · log n)` where `n ≈ d_model`
- Three diagonal multiplications: `O(n)` each
- Total: `O(d_model · log d_model)`, a significant reduction for large models

### Memory Footprint

Storage requirements drop proportionally with the parameter reduction. Standard FFN implementations must maintain two large weight tensors in memory during training and inference. The Hadamard MLP stores only three parameter vectors of size approximately `d_model`, dramatically reducing memory pressure for edge deployments.

## Implementation in Needle 2

The Hadamard MLP integrates directly into the transformer block structure in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), replacing the traditional FFN sub-module after the attention mechanism.

### Block Integration

The `Block` class inserts the Hadamard MLP after attention and normalization:

```python

# needle/model/architecture.py - Block.__call__

class Block(nn.Module):
    ...
    @nn.compact
    def __call__(self, x, mask=None, rope=None, quant=False, engram_kv=None, site_flags=None):
        # Attention mechanism omitted for brevity

        skip = x
        x = ZCRMSNorm(dtype=self.dtype, name="pre_hada_norm")(x)
        # Hadamard MLP replaces traditional FFN

        x = HadamardMLP(self.d_model, self.dtype, name="hadamard_mlp")(x)
        return skip + x

```

### HadamardMLP Implementation

The core implementation (lines 87-103) constructs the transform using JAX operations:

```python

# needle/model/architecture.py - HadamardMLP class

class HadamardMLP(nn.Module):
    d_model: int
    dtype: jnp.dtype = jnp.bfloat16

    @nn.compact
    def __call__(self, x):
        n = 1 << (self.d_model - 1).bit_length()          # Power-of-2 size

        H = _walsh_matrix(n).astype(self.dtype)          # Fixed Hadamard matrix

        d1 = self.param("d1", jinit.ones, (n,)).astype(self.dtype)
        d2 = self.param("d2", jinit.ones, (n,)).astype(self.dtype)
        d3 = self.param("d3", jinit.constant(0.02), (n,)).astype(self.dtype)

        # Pad to power of 2 if necessary

        pad = n - self.d_model
        z = jnp.pad(x, ((0, 0), (0, 0), (0, pad))) if pad else x
        
        # Core Hadamard transforms with diagonal scaling

        z = (d1 * z) @ H                     # First scaling + Hadamard

        z = nn.silu(d2 * z) @ H              # SiLU + second scaling + Hadamard

        return (d3 * z)[..., : self.d_model] # Final scaling and crop

```

### Contrast with Standard FFN

For reference, a conventional implementation would follow this pattern:

```python
class StandardFFN(nn.Module):
    d_model: int
    d_ff: int = None  # Typically 4 * d_model

    dtype: jnp.dtype = jnp.bfloat16

    @nn.compact
    def __call__(self, x):
        if self.d_ff is None:
            self.d_ff = 4 * self.d_model
            
        x = nn.Dense(self.d_ff, dtype=self.dtype)(x)      # Up-projection

        x = nn.silu(x)                                     # Activation

        x = nn.Dense(self.d_model, dtype=self.dtype)(x)   # Down-projection

        return x

```

## Summary

- **The Hadamard MLP replaces dense matrix multiplications with fixed Walsh-Hadamard transforms and learnable diagonal scalings**, reducing parameters from `O(d_model²)` to `O(d_model)`.
- **Computational complexity drops from `O(d_model · d_ff)` to `O(d_model · log d_model)`** due to the fast transform algorithm, making inference significantly faster.
- **Implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** where the `HadamardMLP` class integrates seamlessly into the transformer `Block`, substituting for the traditional two-layer FFN.
- **Memory footprint is minimized** by storing only three parameter vectors rather than two full weight matrices, crucial for the lightweight deployment targets of Needle 2.
- **Non-linearity placement follows standard practice** (SiLU activation) but operates between two orthogonal transforms, preserving signal norm while maintaining expressiveness.

## Frequently Asked Questions

### What is the primary advantage of Hadamard MLP over standard FFN?

The primary advantage is **computational and memory efficiency**. While standard FFNs require storing and multiplying two dense matrices of size `[d_model, 4*d_model]`, the Hadamard MLP uses fixed orthogonal transforms and learns only three diagonal vectors. This reduces both memory bandwidth pressure and floating-point operations during inference, making it ideal for resource-constrained environments without significantly impacting model capacity.

### How does the Walsh-Hadamard transform reduce computational cost?

The Walsh-Hadamard transform can be computed using a **Fast Walsh-Hadamard Transform (FWHT)** algorithm similar to the FFT, operating in `O(n log n)` time rather than the `O(n²)` required by matrix multiplication. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the `_walsh_matrix()` function generates the transform matrix, but the actual implementation leverages efficient matrix multiplication primitives that exploit the recursive structure of Hadamard matrices, avoiding explicit storage of the full `n × n` matrix.

### Does the Hadamard MLP maintain the same model capacity as a standard FFN?

Yes, the architecture maintains **comparable expressive power** despite the parameter reduction. The combination of learnable diagonal scalings (`d1`, `d2`, `d3`) and fixed orthogonal transforms preserves the ability to model complex non-linear relationships. The orthogonal nature of the Hadamard matrix ensures signal norms are preserved, while the three stages of scaling provide sufficient degrees of freedom to approximate the behavior of dense projections in standard FFNs.

### Why does Needle 2 pad the input to power-of-2 dimensions?

The Hadamard MLP requires power-of-2 dimensions because **Walsh-Hadamard matrices are naturally defined for sizes `2^k`**. In the `HadamardMLP.__call__` method, the code calculates `n = 1 << (self.d_model - 1).bit_length()` to find the next power of two, then pads the input if necessary. This padding ensures compatibility with the fast transform implementation while the final slicing operation `[..., : self.d_model]` removes the padding before returning the result, maintaining the original model dimensionality.