What Are the Learned Gating Parameters (a, b, g) in Needle 2? A Deep Dive into Scalar Gating Mechanics

Needle 2 learns three scalar gating parameters—multiplier a, bias b, and attention gate g—that act as soft switches via sigmoid activation to control information flow through pre-attention, post-attention, and residual pathways in each transformer layer.

Needle 2 implements a "Simple-Attention-Network" architecture that relies on learned scalar gating parameters to dynamically regulate transformer computations. These parameters, denoted as a, b, and g, are stored as per-layer scalars and applied through sigmoid-activated gates throughout the model's forward pass according to the cactus-compute/needle source code.

Understanding the Three Learned Gating Parameters (a, b, g)

The Needle 2 transformer uses three distinct scalar values to control its gating mechanisms. Each parameter serves a specific mathematical function in the gating logits before the final sigmoid activation clamps values to the [0, 1] range.

Parameter a – The Multiplier

The a parameter acts as a learned multiplier that scales the gating logits before the sigmoid function is applied. In the source code, these appear as mhc_a_pre, mhc_a_post, and mhc_a_res within the hc dictionary in needle/model/architecture.py (lines approximately 596–604). This scalar directly multiplies the input projection (x @ φ) to control the gate's sensitivity to the input features.

Parameter b – The Bias

The b parameter provides a learned bias term added to the gating logits prior to sigmoid activation. Defined alongside parameter a in the same hc dictionary, variables such as mhc_b_pre, mhc_b_post, and mhc_b_res shift the gating threshold, allowing the model to learn optimal offset values for opening or closing specific pathways independently of the input magnitude.

Parameter g – The Attention Gate

The g parameter represents the attention gate itself, implemented as the scalar attn_gate tensor. Defined in Block._gate() in needle/model/architecture.py (lines approximately 14–16) and applied at line 32 via skip + self._gate("attn_gate") * x, this scalar multiplies the output of the attention block before the residual addition, effectively controlling how much of the attention output reaches the subsequent layers.

Where the Gating Parameters Are Defined in the Source Code

According to the cactus-compute/needle implementation, the gating parameters reside in specific locations within the architecture module:

• needle/model/architecture.py (lines ~596–604): Defines the a and b parameters (mhc_a_pre, mhc_b_pre, etc.) inside the hc dictionary as scalar tensors of shape ().
• needle/model/architecture.py (lines ~14–16 and ~32): Defines and applies the g parameter (attn_gate) within the Block class's _gate() method and the forward pass.
• needle/model/finetune.py: Confirms that these gating parameters are excluded from LoRA target matrices, preserving their learned values during fine-tuning.

How the Gating Flow Works in Needle 2

The learned parameters a, b, and g operate across four distinct gating stages in each transformer layer:

1. Pre-gate: Computes a_pre * (x @ φ_pre) + b_pre + pre_off, applies sigmoid, and scales the input to the Hadamard-MLP.
2. Attention gate: Applies the scalar attn_gate (g) to the attention block output via multiplication before the residual addition.
3. Post-gate: Computes a_post * (x @ φ_post) + b_post + post_off, applies sigmoid, and scales the Hadamard-MLP output.
4. Residual gate: Computes a_res * (x @ φ_res) + b_res, applies sinkhorn normalization, and controls the residual mixing ratio.

All gates use sigmoid activation to ensure soft, differentiable switching between [0, 1], allowing the model to learn continuous gating policies rather than hard binary decisions.

Inspecting and Modifying Gating Parameters in Practice

You can inspect and override these scalar parameters directly from the model checkpoint. Each transformer layer stores its own independent set of scalars:

import needle as nd
import jax.numpy as jnp

# Load the Needle 2 model (the binary engine is fetched automatically)

model, params = nd.load("cactus/needle-2")

# Inspect the scalar gates of the first transformer layer

layer0 = params["layers"]["block_0"]        # block_0 = first layer

a_pre   = layer0["mhc_a_pre"]               # → scalar (a)

b_pre   = layer0["mhc_b_pre"]               # → scalar (b)

g_attn  = layer0["attn_gate"]               # → scalar (g)

print("a_pre:", a_pre)
print("b_pre:", b_pre)
print("g (attention gate):", g_attn)

# Example: manually overriding the attention gate for inference

override_params = params.copy()
override_params["layers"]["block_0"]["attn_gate"] = jnp.array(0.0)  # force gate closed

out = model.apply(override_params, input_ids)   # run with modified gate

Fine-Tuning Behavior with LoRA

When fine-tuning Needle 2 using LoRA (Low-Rank Adaptation), the learned gating parameters a, b, and g remain frozen by default. The needle/model/finetune.py implementation explicitly excludes these scalars from the trainable LoRA targets, ensuring that the learned gating behavior from pre-training remains intact:

# Initialise LoRA fine-tuning – the gating parameters are automatically frozen

finetuned_params = nd.finetune(
    model,
    params,
    train_dataset,
    lora_rank=4,
    freeze=("mhc_a_pre", "mhc_b_pre", "attn_gate")  # keep learned gates intact

)

Summary

• Needle 2 learns three scalar gating parameters per layer: a (multiplier), b (bias), and g (attention gate).
• These parameters are defined in needle/model/architecture.py within the hc dictionary (a and b) and the Block._gate() method (g).
• All gates apply sigmoid activation to produce soft switches in the [0, 1] range.
• The gating flow includes four stages: pre-gate, attention gate, post-gate, and residual gate.
• Parameters are stored as scalars (shape ()) in the checkpoint and remain frozen during LoRA fine-tuning.
• You can inspect and manually override these values using the Needle Python API.

Frequently Asked Questions

What do the learned gating parameters (a, b, g) control in Needle 2?

The parameters control three distinct stages of the Simple-Attention-Network transformer. Parameter a scales gating logits as a multiplier, b provides a learnable bias shift, and g directly gates the attention block output. Together, they regulate the flow of information through pre-attention projections, post-attention mixing, and residual connections via sigmoid-activated soft switches.

Where are the gating parameters stored in the Needle 2 checkpoint?

The gating parameters are stored as scalar tensors of shape () within each layer's parameter dictionary. Specifically, a and b parameters appear as mhc_a_pre, mhc_b_pre, etc., and g appears as attn_gate under params["layers"]["block_{n}"] for each layer index n.

Can I modify the attention gate g during inference?

Yes. Since g (attn_gate) is exposed as a standard scalar parameter in the model dictionary, you can override its value using jax.numpy arrays before calling model.apply(). Setting attn_gate to 0.0 effectively closes the gate, while 1.0 forces it fully open, bypassing the learned sigmoid activation for that specific forward pass.

Why are the gating parameters frozen during LoRA fine-tuning?

The gating parameters remain frozen because they represent fundamental architectural decisions learned during pre-training that control the model's computation graph structure. According to needle/model/finetune.py, LoRA targets only the low-rank adaptation matrices, leaving scalars like a, b, and g intact to preserve the learned gating policies while adapting the model to new tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}