How Needle 2 Implements Confidence Scoring: The ConfidenceHead Architecture

Needle 2 computes confidence scores using a dedicated ConfidenceHead module that pools transformer hidden states through eight learned probes and projects the result to a scalar logit.

The cactus-compute/needle repository implements a deterministic confidence estimation mechanism directly within its transformer stack. Understanding confidence scoring in Needle 2 requires analyzing the three-stage pipeline defined in the ConfidenceHead class and its integration with the encoder's hidden states. This system produces calibrated scalar logits during inference to quantify model certainty.

Architecture of the ConfidenceHead Module

The confidence scoring mechanism resides in needle/model/architecture.py and operates as a distinct head on top of the transformer encoder. Unlike the main generation pathway, the ConfidenceHead implements a specialized forward pass called forward_confidence that processes frozen encoder representations.

Step 1: Encoding Hidden Cells with Gradient Stopping

The process begins when forward_confidence extracts representations from the transformer encoder. At lines 72-75, the code invokes hidden_cells within a jax.lax.stop_gradient block to freeze the encoder weights during confidence computation:

cells = jax.lax.stop_gradient(
    self.hidden_cells(tokens, quant=quant, window=window, sink=sink))

This gradient stopping ensures that confidence training or inference does not backpropagate into the core transformer weights.

Step 2: Probe Pooling with Learned Parameters

The variable-length sequence of hidden cells is collapsed into a fixed-size vector via probe pooling. The ConfidenceHead initializes eight learned probes and applies them through the probe_pool function (lines 68-73):

pooled = probe_pool(cells,
                    self.param("probes", default_init(),
                               (self.PROBES, cells.shape[-1])),
                    keep, self.dtype)

The self.PROBES constant defines eight distinct projection vectors that learn to aggregate relevant signals from across the token sequence.

Step 3: Scalar Logit Projection

Finally, the pooled representation passes through a single-output dense layer to produce the raw confidence score. Lines 73-76 show the projection cast to float32 for numerical stability:

logit = nn.Dense(1, dtype=self.dtype, use_bias=True,
                 kernel_init=default_init(), name="proj")(pooled)
return logit[..., 0].astype(jnp.float32)

The ConfidenceHead is instantiated within the SimpleAttentionNetwork class at line 88:

self.confidence_head = ConfidenceHead(cfg.jax_dtype)

Runtime API and Finetuned Model Handling

The high-level Python interface in needle/__init__.py manages how these raw logits reach the end user and handles edge cases for custom weights.

Accessing Confidence Scores via Needle.complete

When calling Needle.complete, the response envelope includes a "confidence" field containing the scalar logit from the architecture's forward pass. This value represents the model's raw certainty regarding the generated completion.

Disabling Confidence for Finetuned Weights

If the engine loads custom weights through the weights parameter, the confidence head is deliberately disabled to prevent uncalibrated outputs. Lines 60-64 and 124-125 check for finetuned weights and set the field to None:

if self._weights:
    response["confidence"] = None

The wrapper emits a warning because the ConfidenceHead probes and projection layer are not updated during standard finetuning procedures.

Code Examples

Retrieving Confidence with the Base Model

To obtain a confidence score using the default pretrained weights:

from needle import Needle

agent = Needle()
response = agent.complete("Is the Earth round?")
print(response["confidence"])      # → e.g., 0.93 (raw logit)

Multi-Step Interactions

The run method propagates confidence values through multi-step reasoning chains:

result = agent.run(
    "Summarize the following article and rate its credibility.", 
    max_steps=3)
print(result["confidence"])        # → confidence for the final answer

Handling Finetuned Checkpoints

When loading finetuned weights, expect None values and appropriate warnings:

agent = Needle(weights="my_finetuned.cact")
resp = agent.complete("Is this claim true?")
print(resp["confidence"])          # → None (un-calibrated)

Summary

• Three-stage pipeline: Confidence scoring in Needle 2 extracts frozen hidden cells via forward_confidence, pools them via eight learned probes, and projects to a scalar logit in needle/model/architecture.py.
• API integration: The Needle.complete method surfaces this value under the "confidence" key in the response JSON returned to users.
• Finetuning limitations: Custom weights disable the confidence head (returning None in needle/__init__.py lines 60-64) because the probes and projection layer lack calibration for the new weights.

Frequently Asked Questions

Where is the ConfidenceHead class implemented?

The ConfidenceHead class is defined in needle/model/architecture.py. This file contains the forward_confidence method, the probe pooling logic, and the scalar projection layer used to generate confidence scores.

Why does Needle 2 return None for confidence with finetuned models?

When loading finetuned weights via the weights parameter, the wrapper in needle/__init__.py lines 60-64 sets the confidence field to None. The confidence head is not updated during standard finetuning procedures, rendering its outputs unreliable without recalibration on the new weight distribution.

How many probes does the pooling mechanism use?

The architecture uses 8 learned probes to pool the hidden cell representations. This is defined by the self.PROBES parameter in the ConfidenceHead class and applied through the probe_pool function at lines 68-73.

What data type is the final confidence score?

The final logit is explicitly cast to float32 before being returned to the user (line 76). This ensures consistent numerical precision across different JAX configurations and hardware backends.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}