# What Signals Are Used to Determine the Confidence Score of a Needle Agent's Call?

> Learn how Needle determines a confidence score for agent calls. Discover the specific signals and mechanisms used, including pooled hidden-state embeddings and the ConfidenceHead module.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-31

---

**The confidence score of a Needle agent's call is determined solely by the pooled hidden-state embeddings from the transformer's encoder, passed through a learned linear projection in the `ConfidenceHead` module.**

The Needle framework, developed by cactus-compute, generates confidence scores to indicate how certain the model is about its generated responses. Understanding these internal signals helps developers interpret model outputs and build more reliable AI systems.

## The ConfidenceHead Module Architecture

The confidence computation centers on the `ConfidenceHead` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This module processes internal model representations to produce a single scalar value representing confidence.

### Step 1: Hidden State Extraction

The transformer encoder produces a tensor of hidden states referred to as `cells`. These cells capture the model's processed understanding of the input tokens after self-attention and feed-forward transformations.

### Step 2: Probe Pooling Aggregation

```python
from needle.model.architecture import probe_pool

# cells: [batch, sequence_length, hidden_dim]

pooled = probe_pool(cells, probes=self.PROBES)  # PROBES = 8

```

The `probe_pool` routine collapses the sequence of hidden cells into **8 learned probe vectors**. This pooling mechanism aggregates distributed information across the entire sequence into a compact representation that the confidence head can evaluate.

### Step 3: Linear Projection to Confidence Logit

The pooled representation feeds through a single-output dense layer:

```python

# Inside ConfidenceHead.__call__

self.dense = nn.Dense(1, dtype=self.dtype, kernel_init=default_init())
logit = self.dense(pooled)  # [batch, 1] → scalar

```

The learned weights of this `nn.Dense(1, ...)` layer determine how the pooled features map to a confidence value. No external features, input metadata, or output token probabilities contribute to this score.

### Step 4: Response Integration

The scalar logit converts to a Python `float` and populates the `"confidence"` key in the agent's JSON response, as implemented in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py).

## Complete Forward Confidence Flow

The full confidence computation occurs in `SimpleAttentionNetwork.forward_confidence` at lines 572–576 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py):

```python
from needle.model.architecture import SimpleAttentionNetwork

# model: initialized SimpleAttentionNetwork

# tokens: encoded input token IDs

logits, confidence = model.forward_confidence(tokens, quant=False)

# confidence: scalar float derived from hidden states only

```

This method returns both the output logits for token generation and the confidence score computed from the encoder's hidden states.

## Practical Code Examples

### Direct Confidence Head Usage

```python
import jax.numpy as jnp
from needle.model.architecture import ConfidenceHead, probe_pool, default_init

# Simulate hidden states from transformer encoder

batch, seq_len, hidden_dim = 1, 512, 1024
cells = jnp.ones((batch, seq_len, hidden_dim), dtype=jnp.bfloat16)

# Initialize and run confidence head

conf_head = ConfidenceHead(dtype=jnp.bfloat16)
confidence_logit = conf_head(cells)

print(f"Confidence score: {float(confidence_logit):.4f}")

```

### Agent-Level API Access

```python
from needle import Needle

agent = Needle()
response = agent.run("Explain the transformer architecture.")

print(f"Answer: {response['answer']}")
print(f"Confidence: {response['confidence']:.4f}")

```

### Custom Inference Pipeline

```python
from needle.model.architecture import SimpleAttentionNetwork

# Load model weights and tokenizer

model = SimpleAttentionNetwork(...)
tokens = tokenizer.encode("What signals determine confidence?")

# Explicit confidence extraction

_, confidence = model.forward_confidence(tokens, quant=False)

# Returns: confidence logit from pooled hidden states

```

## Key Files and Their Roles

| File | Relevant Lines | Purpose |
|------|---------------|---------|
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | 63–76 | `ConfidenceHead` class definition with pooling and dense projection |
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | 572–576 | `forward_confidence` method connecting encoder to confidence head |
| [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) | — | Registers confidence head in `HEAD_CODES` for model serialization |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | — | Injects confidence value into agent response dictionary |

## Summary

- **Primary signal**: Pooled hidden-state embeddings (`probe_pool` output with 8 learned probes)
- **Learned component**: Single Dense layer weights projecting pooled features to scalar logit
- **Excluded signals**: No input text features, no output token probabilities, no external metadata
- **Source architecture**: `ConfidenceHead` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)

The confidence score is entirely a function of internal transformer representations, making it a measure of the model's self-assessment of its encoded understanding rather than output quality or input characteristics.

## Frequently Asked Questions

### Does the confidence score use the model's output tokens or only internal states?

The confidence score uses **only internal hidden states**, not output tokens. The `ConfidenceHead` receives pooled embeddings from the encoder before or parallel to token generation. No token probabilities or generated sequences influence the score.

### What does the `probe_pool` function do and why 8 probes?

The `probe_pool` function aggregates variable-length sequences into fixed representations using **8 learned probe vectors**. This number (defined as `self.PROBES = 8`) provides sufficient capacity to capture diverse aspects of model state while remaining computationally efficient.

### Can the confidence score be calibrated or fine-tuned?

Yes, the score depends on **learned weights** in the final Dense layer. Fine-tuning the `ConfidenceHead` specifically, or the full model with confidence-aware objectives, would recalibrate how hidden states map to confidence values without changing the underlying signal source.

### Where is the confidence value exposed to end users?

The confidence value appears in the agent's JSON response under the `"confidence"` key, populated in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py). Applications can access it via `agent.run(query)["confidence"]` or extract it from raw model outputs through `forward_confidence`.