# What Are the Learned Gating Parameters (a, b, g) in Needle 2? A Deep Dive into Scalar Gating Mechanics

> Explore Needle 2's learned gating parameters a, b, and g. Understand how these scalar values control information flow and attention pathways in transformer layers. Dive deep into scalar gating mechanics.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-11

---

**Needle 2 learns three scalar gating parameters—multiplier a, bias b, and attention gate g—that act as soft switches via sigmoid activation to control information flow through pre-attention, post-attention, and residual pathways in each transformer layer.**

Needle 2 implements a "Simple-Attention-Network" architecture that relies on learned scalar gating parameters to dynamically regulate transformer computations. These parameters, denoted as **a**, **b**, and **g**, are stored as per-layer scalars and applied through sigmoid-activated gates throughout the model's forward pass according to the cactus-compute/needle source code.

## Understanding the Three Learned Gating Parameters (a, b, g)

The Needle 2 transformer uses three distinct scalar values to control its gating mechanisms. Each parameter serves a specific mathematical function in the gating logits before the final sigmoid activation clamps values to the `[0, 1]` range.

### Parameter a – The Multiplier

The **a** parameter acts as a learned multiplier that scales the gating logits before the sigmoid function is applied. In the source code, these appear as `mhc_a_pre`, `mhc_a_post`, and `mhc_a_res` within the `hc` dictionary in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines approximately 596–604). This scalar directly multiplies the input projection (`x @ φ`) to control the gate's sensitivity to the input features.

### Parameter b – The Bias

The **b** parameter provides a learned bias term added to the gating logits prior to sigmoid activation. Defined alongside parameter **a** in the same `hc` dictionary, variables such as `mhc_b_pre`, `mhc_b_post`, and `mhc_b_res` shift the gating threshold, allowing the model to learn optimal offset values for opening or closing specific pathways independently of the input magnitude.

### Parameter g – The Attention Gate

The **g** parameter represents the attention gate itself, implemented as the scalar `attn_gate` tensor. Defined in `Block._gate()` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines approximately 14–16) and applied at line 32 via `skip + self._gate("attn_gate") * x`, this scalar multiplies the output of the attention block before the residual addition, effectively controlling how much of the attention output reaches the subsequent layers.

## Where the Gating Parameters Are Defined in the Source Code

According to the cactus-compute/needle implementation, the gating parameters reside in specific locations within the architecture module:

- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** (lines ~596–604): Defines the **a** and **b** parameters (`mhc_a_pre`, `mhc_b_pre`, etc.) inside the `hc` dictionary as scalar tensors of shape `()`.
- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** (lines ~14–16 and ~32): Defines and applies the **g** parameter (`attn_gate`) within the `Block` class's `_gate()` method and the forward pass.
- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)**: Confirms that these gating parameters are excluded from LoRA target matrices, preserving their learned values during fine-tuning.

## How the Gating Flow Works in Needle 2

The learned parameters **a**, **b**, and **g** operate across four distinct gating stages in each transformer layer:

1. **Pre-gate**: Computes `a_pre * (x @ φ_pre) + b_pre + pre_off`, applies sigmoid, and scales the input to the Hadamard-MLP.
2. **Attention gate**: Applies the scalar `attn_gate` (**g**) to the attention block output via multiplication before the residual addition.
3. **Post-gate**: Computes `a_post * (x @ φ_post) + b_post + post_off`, applies sigmoid, and scales the Hadamard-MLP output.
4. **Residual gate**: Computes `a_res * (x @ φ_res) + b_res`, applies sinkhorn normalization, and controls the residual mixing ratio.

All gates use sigmoid activation to ensure soft, differentiable switching between `[0, 1]`, allowing the model to learn continuous gating policies rather than hard binary decisions.

## Inspecting and Modifying Gating Parameters in Practice

You can inspect and override these scalar parameters directly from the model checkpoint. Each transformer layer stores its own independent set of scalars:

```python
import needle as nd
import jax.numpy as jnp

# Load the Needle 2 model (the binary engine is fetched automatically)

model, params = nd.load("cactus/needle-2")

# Inspect the scalar gates of the first transformer layer

layer0 = params["layers"]["block_0"]        # block_0 = first layer

a_pre   = layer0["mhc_a_pre"]               # → scalar (a)

b_pre   = layer0["mhc_b_pre"]               # → scalar (b)

g_attn  = layer0["attn_gate"]               # → scalar (g)

print("a_pre:", a_pre)
print("b_pre:", b_pre)
print("g (attention gate):", g_attn)

# Example: manually overriding the attention gate for inference

override_params = params.copy()
override_params["layers"]["block_0"]["attn_gate"] = jnp.array(0.0)  # force gate closed

out = model.apply(override_params, input_ids)   # run with modified gate

```

## Fine-Tuning Behavior with LoRA

When fine-tuning Needle 2 using LoRA (Low-Rank Adaptation), the learned gating parameters **a**, **b**, and **g** remain frozen by default. The [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) implementation explicitly excludes these scalars from the trainable LoRA targets, ensuring that the learned gating behavior from pre-training remains intact:

```python

# Initialise LoRA fine-tuning – the gating parameters are automatically frozen

finetuned_params = nd.finetune(
    model,
    params,
    train_dataset,
    lora_rank=4,
    freeze=("mhc_a_pre", "mhc_b_pre", "attn_gate")  # keep learned gates intact

)

```

## Summary

- **Needle 2** learns three scalar gating parameters per layer: **a** (multiplier), **b** (bias), and **g** (attention gate).
- These parameters are defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) within the `hc` dictionary (**a** and **b**) and the `Block._gate()` method (**g**).
- All gates apply sigmoid activation to produce soft switches in the `[0, 1]` range.
- The gating flow includes four stages: pre-gate, attention gate, post-gate, and residual gate.
- Parameters are stored as scalars (shape `()`) in the checkpoint and remain frozen during LoRA fine-tuning.
- You can inspect and manually override these values using the Needle Python API.

## Frequently Asked Questions

### What do the learned gating parameters (a, b, g) control in Needle 2?

The parameters control three distinct stages of the Simple-Attention-Network transformer. Parameter **a** scales gating logits as a multiplier, **b** provides a learnable bias shift, and **g** directly gates the attention block output. Together, they regulate the flow of information through pre-attention projections, post-attention mixing, and residual connections via sigmoid-activated soft switches.

### Where are the gating parameters stored in the Needle 2 checkpoint?

The gating parameters are stored as scalar tensors of shape `()` within each layer's parameter dictionary. Specifically, **a** and **b** parameters appear as `mhc_a_pre`, `mhc_b_pre`, etc., and **g** appears as `attn_gate` under `params["layers"]["block_{n}"]` for each layer index `n`.

### Can I modify the attention gate g during inference?

Yes. Since **g** (`attn_gate`) is exposed as a standard scalar parameter in the model dictionary, you can override its value using `jax.numpy` arrays before calling `model.apply()`. Setting `attn_gate` to `0.0` effectively closes the gate, while `1.0` forces it fully open, bypassing the learned sigmoid activation for that specific forward pass.

### Why are the gating parameters frozen during LoRA fine-tuning?

The gating parameters remain frozen because they represent fundamental architectural decisions learned during pre-training that control the model's computation graph structure. According to [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), LoRA targets only the low-rank adaptation matrices, leaving scalars like **a**, **b**, and **g** intact to preserve the learned gating policies while adapting the model to new tasks.