# Why Needle 2 Confidence Scores Fail After Fine-Tuning: The Calibration Head Problem

> Discover why Needle 2 confidence scores fail after fine-tuning. Learn how the frozen ConfidenceHead network causes calibration issues and affects model distribution.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-25

---

**Fine-tuning Needle 2 disables confidence scores because the `ConfidenceHead` network is explicitly frozen during training, leaving it uncalibrated for the new model distribution.**

When you fine-tune Needle 2, the model's internal **confidence head**—a separate neural network that produces scalar confidence logits—remains completely untrained. This architectural decision, implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), prevents the confidence scores from aligning with your fine-tuned model's output distribution. Rather than expose misleading values, the Needle library intentionally disables confidence reporting entirely.

---

## How the Confidence Head Works in Needle 2

The `ConfidenceHead` is a small auxiliary network defined at lines 63–76 of [[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L63-L76):

```python

# Architecture excerpt (needle/model/architecture.py)

class ConfidenceHead(nn.Module):
    """Produces a scalar logit representing model confidence."""
    def __init__(self, hidden_dim: int):
        super().__init__()
        self.project = nn.Linear(hidden_dim, 1)
    
    def forward(self, hidden_states: Tensor) -> Tensor:
        # Confidence derived from final layer representations

        return self.project(hidden_states[:, -1, :]).squeeze(-1)

```

This head is trained during **pre-training only**, learning to map final-layer hidden states to well-calibrated confidence estimates. Once pre-training completes, the head's weights are fixed for all downstream use cases.

---

## Why Fine-Tuning Breaks Calibration

Fine-tuning in Needle 2 follows a **selective parameter update strategy**. The training loop, located in [[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), optimizes only the main transformer layers while explicitly excluding the confidence head from the optimizer's parameter groups.

This creates a fundamental mismatch:

| Component | Updated During Fine-Tuning? | Effect on Confidence |
|-----------|-----------------------------|----------------------|
| Transformer layers | **Yes** | Logit distribution shifts |
| Classification head | **Yes** | Task-specific predictions adapt |
| `ConfidenceHead` | **No** | Confidence estimates become uncalibrated |

The confidence head continues to interpret hidden states using its pre-trained mapping, but those hidden states now occupy a different region of representation space. The result: confidence scores that appear plausible but systematically misestimate true model accuracy.

---

## How Needle 2 Disables Unreliable Confidence Scores

To prevent users from acting on miscalibrated values, Needle 2 implements a two-layer safety mechanism in [[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py):

### Constructor Warning (Lines 60–63)

```python

# From needle/__init__.py

if weights_path != "base.cact" and not weights_path.startswith("needle/"):
    warnings.warn(
        "finetuning does not update the confidence head, so scores are uncalibrated "
        "for tuned weights; this agent reports confidence as None",
        UserWarning,
        stacklevel=2
    )

```

This triggers whenever you load non-base weights, alerting users to the confidence limitation before any inference occurs.

### Runtime Enforcement (Lines 24–25)

```python

# From needle/__init__.py response construction

if self._using_finetuned_weights:
    response["confidence"] = None  # Force disable for tuned checkpoints

```

The response building code unconditionally sets `confidence` to `None` for fine-tuned models, ensuring no uncalibrated value can propagate to downstream systems.

---

## Observing the Behavior: Code Examples

### Base Model: Confidence Available

```python
from needle import Needle

# Load official pre-trained weights

agent = Needle(weights="base.cact")
response = agent.complete("Explain quantum computing in one sentence")

print(response["confidence"])  # → 0.87 (well-calibrated probability)

```

### Fine-Tuned Model: Confidence Disabled

```python
from needle import Needle

# Load your fine-tuned checkpoint

finetuned_agent = Needle(weights="my_finetuned.cact")

# UserWarning emitted here about uncalibrated confidence head

response = finetuned_agent.complete("Explain quantum computing in one sentence")
print(response["confidence"])  # → None (explicitly disabled)

```

The fine-tuning CLI also surfaces this limitation. At lines 401–402 of [[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py#L401-L402), training completion prints:

```

note: confidence reports None with tuned weights; the head is not tuned

```

---

## Can You Restore Confidence Scores After Fine-Tuning?

Currently, Needle 2 offers **no built-in mechanism** to recalibrate or retrain the confidence head post fine-tuning. The architecture separates head training from the main fine-tuning loop by design, prioritizing stable task adaptation over auxiliary metric preservation.

Potential workarounds require modifying the training infrastructure:

- **Option 1:** Extend [`finetune.py`](https://github.com/cactus-compute/needle/blob/main/finetune.py) to include `ConfidenceHead` parameters in the optimizer with a small learning rate
- **Option 2:** Implement post-hoc calibration using temperature scaling on a held-out validation set
- **Option 3:** Train a separate confidence model on frozen fine-tuned representations

None of these are officially supported according to the source code in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md).

---

## Summary

- **Root cause:** The `ConfidenceHead` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) is frozen during fine-tuning, creating distribution mismatch with updated transformer layers
- **Safety mechanism:** [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) emits warnings and forces `confidence = None` for all fine-tuned checkpoints
- **CLI transparency:** [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) confirms this behavior in training output
- **Current limitation:** No supported pathway exists to recover calibrated confidence scores after fine-tuning Needle 2

---

## Frequently Asked Questions

### Why does Needle 2 disable confidence instead of showing uncalibrated scores?

The library prioritizes **avoiding misleading signals** over providing noisy metrics. Uncalibrated confidence scores can lead to harmful downstream decisions—such as automated filtering or threshold-based routing—when users mistakenly trust them. Setting `confidence = None` makes the limitation explicit and forces intentional handling.

### Can I modify the source code to train the confidence head during fine-tuning?

Yes, but this requires changing the parameter groups in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) to include `model.confidence_head.parameters()`. Note that this may destabilize fine-tuning if the confidence head's gradients compete with task objectives; no validation testing is included in the repository for this configuration.

### How can I estimate model confidence without the built-in score?

Consider **ensemble disagreement** across multiple outputs with temperature-adjusted sampling, or implement **token-level entropy** over the output distribution as a proxy. These heuristic approaches lack the head's learned calibration but provide rough uncertainty quantification suitable for monitoring.

### Does this limitation apply to all Needle 2 weight formats?

The check in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) specifically tests for paths not matching `"base.cact"` or `"needle/"` prefix. Official base weights and internal `needle/` package weights retain confidence functionality. Any user-supplied checkpoint—including merged or converted formats—triggers the disable logic.