# How Needle 2 Implements Confidence Scoring: The ConfidenceHead Architecture

> Needle 2 uses its ConfidenceHead architecture to compute confidence scores by pooling transformer hidden states via learned probes and projecting to a scalar logit. Learn how it works.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-24

---

**Needle 2 computes confidence scores using a dedicated `ConfidenceHead` module that pools transformer hidden states through eight learned probes and projects the result to a scalar logit.**

The **cactus-compute/needle** repository implements a deterministic confidence estimation mechanism directly within its transformer stack. Understanding **confidence scoring in Needle 2** requires analyzing the three-stage pipeline defined in the `ConfidenceHead` class and its integration with the encoder's hidden states. This system produces calibrated scalar logits during inference to quantify model certainty.

## Architecture of the ConfidenceHead Module

The confidence scoring mechanism resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and operates as a distinct head on top of the transformer encoder. Unlike the main generation pathway, the `ConfidenceHead` implements a specialized forward pass called `forward_confidence` that processes frozen encoder representations.

### Step 1: Encoding Hidden Cells with Gradient Stopping

The process begins when `forward_confidence` extracts representations from the transformer encoder. At lines 72-75, the code invokes `hidden_cells` within a `jax.lax.stop_gradient` block to freeze the encoder weights during confidence computation:

```python
cells = jax.lax.stop_gradient(
    self.hidden_cells(tokens, quant=quant, window=window, sink=sink))

```

This gradient stopping ensures that confidence training or inference does not backpropagate into the core transformer weights.

### Step 2: Probe Pooling with Learned Parameters

The variable-length sequence of hidden cells is collapsed into a fixed-size vector via **probe pooling**. The `ConfidenceHead` initializes eight learned probes and applies them through the `probe_pool` function (lines 68-73):

```python
pooled = probe_pool(cells,
                    self.param("probes", default_init(),
                               (self.PROBES, cells.shape[-1])),
                    keep, self.dtype)

```

The `self.PROBES` constant defines eight distinct projection vectors that learn to aggregate relevant signals from across the token sequence.

### Step 3: Scalar Logit Projection

Finally, the pooled representation passes through a single-output dense layer to produce the raw confidence score. Lines 73-76 show the projection cast to `float32` for numerical stability:

```python
logit = nn.Dense(1, dtype=self.dtype, use_bias=True,
                 kernel_init=default_init(), name="proj")(pooled)
return logit[..., 0].astype(jnp.float32)

```

The `ConfidenceHead` is instantiated within the `SimpleAttentionNetwork` class at line 88:

```python
self.confidence_head = ConfidenceHead(cfg.jax_dtype)

```

## Runtime API and Finetuned Model Handling

The high-level Python interface in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) manages how these raw logits reach the end user and handles edge cases for custom weights.

### Accessing Confidence Scores via Needle.complete

When calling `Needle.complete`, the response envelope includes a `"confidence"` field containing the scalar logit from the architecture's forward pass. This value represents the model's raw certainty regarding the generated completion.

### Disabling Confidence for Finetuned Weights

If the engine loads custom weights through the `weights` parameter, the confidence head is deliberately disabled to prevent uncalibrated outputs. Lines 60-64 and 124-125 check for finetuned weights and set the field to `None`:

```python
if self._weights:
    response["confidence"] = None

```

The wrapper emits a warning because the `ConfidenceHead` probes and projection layer are not updated during standard finetuning procedures.

## Code Examples

### Retrieving Confidence with the Base Model

To obtain a confidence score using the default pretrained weights:

```python
from needle import Needle

agent = Needle()
response = agent.complete("Is the Earth round?")
print(response["confidence"])      # → e.g., 0.93 (raw logit)

```

### Multi-Step Interactions

The `run` method propagates confidence values through multi-step reasoning chains:

```python
result = agent.run(
    "Summarize the following article and rate its credibility.", 
    max_steps=3)
print(result["confidence"])        # → confidence for the final answer

```

### Handling Finetuned Checkpoints

When loading finetuned weights, expect `None` values and appropriate warnings:

```python
agent = Needle(weights="my_finetuned.cact")
resp = agent.complete("Is this claim true?")
print(resp["confidence"])          # → None (un-calibrated)

```

## Summary

- **Three-stage pipeline**: Confidence scoring in Needle 2 extracts frozen hidden cells via `forward_confidence`, pools them via eight learned probes, and projects to a scalar logit in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- **API integration**: The `Needle.complete` method surfaces this value under the `"confidence"` key in the response JSON returned to users.
- **Finetuning limitations**: Custom weights disable the confidence head (returning `None` in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) lines 60-64) because the probes and projection layer lack calibration for the new weights.

## Frequently Asked Questions

### Where is the ConfidenceHead class implemented?

The `ConfidenceHead` class is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This file contains the `forward_confidence` method, the probe pooling logic, and the scalar projection layer used to generate confidence scores.

### Why does Needle 2 return None for confidence with finetuned models?

When loading finetuned weights via the `weights` parameter, the wrapper in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) lines 60-64 sets the confidence field to `None`. The confidence head is not updated during standard finetuning procedures, rendering its outputs unreliable without recalibration on the new weight distribution.

### How many probes does the pooling mechanism use?

The architecture uses **8 learned probes** to pool the hidden cell representations. This is defined by the `self.PROBES` parameter in the `ConfidenceHead` class and applied through the `probe_pool` function at lines 68-73.

### What data type is the final confidence score?

The final logit is explicitly cast to `float32` before being returned to the user (line 76). This ensures consistent numerical precision across different JAX configurations and hardware backends.