# How Needle 2 Provides Calibrated Confidence Scores: Architecture and Implementation

> Discover how Needle 2 architects its ConfidenceHead for calibrated confidence scores. Learn about its implementation and why finetuned weights disable output to ensure accuracy.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-19

---

**Needle 2 generates calibrated confidence scores through a dedicated `ConfidenceHead` neural module that outputs a scalar value aligned with actual success rates, but only when using base model weights—finetuned weights automatically disable confidence output to prevent miscalibration.**

Needle 2, the second-generation model from the `cactus-compute/needle` repository, introduces a specialized mechanism for expressing uncertainty in generated responses. Unlike standard language models that rely on heuristic confidence measures, Needle 2 implements a learned confidence head trained jointly with the base model to produce statistically calibrated scores. This article examines the architecture, training process, and practical implementation of Needle 2 calibrated confidence scores based on the actual source code.

## The ConfidenceHead Architecture

### Core Module Implementation

At the heart of Needle 2's calibration system lies the **`ConfidenceHead`**, a lightweight neural module defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 63–76). This component pools the transformer's hidden states, applies a learned linear projection, and outputs a single logit representing the model's certainty.

The head is designed to be computationally efficient, minimizing overhead during inference while providing meaningful uncertainty estimates for every generated token sequence.

### The forward_confidence Method

The primary interface for obtaining confidence values is the **`forward_confidence`** method, located in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 72–77). This method performs three critical operations:

1. **Stops gradients** on the hidden cells to prevent calibration drift during inference
2. **Builds a keep mask** that ignores padding tokens, ensuring confidence is computed only over meaningful content
3. **Feeds the masked representation** into the `ConfidenceHead` to produce the final scalar value

### Export Integration

For model serialization and runtime routing, the system uses the **`HEAD_CODES`** tuple defined in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (line 289). This mapping assigns a numeric identifier to the confidence head, enabling the inference engine to route confidence queries to the correct module when loading exported models.

## Calibration Process and Training Dynamics

### Training Phase (Base Weights)

During the initial training phase, the confidence head is learned **jointly** with the language modeling and contrastive objectives. This joint training ensures the output logit is statistically aligned with actual success rates—meaning a confidence score of 0.90 genuinely indicates a 90% probability of correctness.

The base weights distributed with Needle 2 retain this calibration because the confidence head was optimized alongside the transformer backbone across the full training distribution.

### Finetuning Phase (Frozen Head)

When users supply custom finetuned weights, the calibration dynamics change significantly. According to [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 58–62), the finetuning script **freezes the confidence head** and updates only the language-model and contrastive heads. Consequently:

- The confidence values would become uncalibrated for the new task distribution
- The `Needle` class constructor detects finetuned weights and emits a runtime warning
- The library forces `response["confidence"] = None` to avoid presenting misleading scores

This design choice explicitly prevents the deployment of uncalibrated confidence estimates when the underlying task distribution has shifted through finetuning.

## Implementing Calibrated Confidence in Practice

### Retrieving Confidence with Base Weights

When using the original (un-finetuned) weights, calibrated confidence scores are automatically available through the Python wrapper:

```python
from needle import Needle

# Initialize with default base weights

agent = Needle()

# Execute query and retrieve confidence

resp = agent.run("What is the capital of France?", max_steps=1)

print("Answer:", resp["output"])
print("Confidence:", resp["confidence"])   # e.g., 0.92 (calibrated)

```

The `run` method internally invokes `forward_confidence` on the model. Because no custom weights are supplied, the confidence head remains active and returns its calibrated scalar value.

### Handling Finetuned Weights (Disabled Confidence)

Loading finetuned checkpoints automatically suppresses confidence output to maintain statistical integrity:

```python
from needle import Needle

# Load custom finetuned checkpoint

finetuned_weights = "my_finetuned_model.cact"
agent = Needle(weights=finetuned_weights)

resp = agent.run("Summarize quantum computing research.", max_steps=2)

print("Answer:", resp["output"])
print("Confidence:", resp["confidence"])   # -> None (head not calibrated)

```

The wrapper warns at construction time that the confidence head was not updated during finetuning, ensuring users understand why the field returns `None`.

### Direct Confidence Head Access

For advanced use cases requiring raw logits, you can invoke the confidence head directly:

```python
import jax.numpy as jnp
from needle.model.architecture import SimpleAttentionNetwork, TransformerConfig

# Initialize model (config parameters omitted for brevity)

config = TransformerConfig(...)
model = SimpleAttentionNetwork(config)

# Tokenize input

tokens = tokenizer.encode("Who wrote 'Pride and Prejudice'?")

# Obtain raw confidence score

conf_score = model.forward_confidence(jnp.array([tokens]))
print("Raw confidence logit:", conf_score)

```

The `forward_confidence` method applies an internal sigmoid activation, returning a float in the range 0–1 suitable for probabilistic interpretation.

## Summary

- **Needle 2** implements calibrated confidence through a dedicated `ConfidenceHead` module in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) that pools hidden states and outputs a calibrated scalar.
- **Calibration is preserved only with base weights**, as the confidence head is trained jointly with the language model during the initial training phase.
- **Finetuned weights disable confidence output** by returning `None`, preventing the deployment of uncalibrated scores when the task distribution shifts.
- The **`forward_confidence`** method provides the public API for confidence retrieval, handling gradient stopping and padding mask application automatically.
- The **`HEAD_CODES`** mapping in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) ensures proper runtime routing of confidence queries in exported models.

## Frequently Asked Questions

### What makes Needle 2 confidence scores "calibrated"?

Needle 2 confidence scores are calibrated because the `ConfidenceHead` is trained jointly with the base model on the full training distribution, ensuring its output logits statistically match actual success rates. A score of 0.80 indicates an 80% empirical probability of correctness, unlike heuristic confidence measures that may systematically overestimate or underestimate uncertainty.

### Why does Needle 2 disable confidence scores for finetuned models?

The library disables confidence output for finetuned models because the finetuning process updates only the language modeling and contrastive heads while **freezing the confidence head** (as implemented in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)). Since the confidence head was not retrained on the new task distribution, its outputs would be statistically uncalibrated. Returning `None` prevents users from acting on potentially misleading certainty estimates.

### How can I access the raw confidence logit directly?

You can access the raw confidence value by calling `model.forward_confidence(jnp.array([tokens]))` on a `SimpleAttentionNetwork` instance from [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This method returns a sigmoid-activated float between 0 and 1. Note that this direct access bypasses the wrapper's finetuning checks, so you must manually verify you are using base weights to obtain calibrated values.

### Where is the confidence head defined in the source code?

The confidence head is defined as the **`ConfidenceHead`** class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 63–76). The inference logic resides in the **`forward_confidence`** method (lines 72–77), while export mappings are handled in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (line 289) via the `HEAD_CODES` tuple.