# Why Confidence Reports None After Fine-Tuning a Needle Model

> Discover why confidence reports None after fine-tuning a Needle model. Learn how an uncalibrated confidence head affects your model's performance and how to address it.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-10

---

**Needle deliberately sets `response["confidence"]` to `None` after fine-tuning because the confidence head—a separate calibration network—is not updated during the fine-tuning process, leaving it uncalibrated for the new weights.**

When working with the `cactus-compute/needle` repository, developers often encounter a sudden loss of confidence scores after fine-tuning a model. This behavior is intentional and protects against misleading predictions. The library intentionally withholds confidence values when it detects that a model has been fine-tuned to prevent uncalibrated outputs from affecting downstream decision-making.

## The Root Cause: Untouched Confidence Head

Fine-tuning in Needle updates only the **main transformer weights** (the language model backbone) while leaving the **confidence head** completely unchanged. The confidence head is a small auxiliary network that converts hidden-state statistics into calibrated probability scores. Because this head remains frozen at its pre-trained weights during fine-tuning, it becomes misaligned with the newly tuned language model.

In [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), line 443 contains a comment documenting this design decision:

> *“confidence reports None with tuned weights; the head is not tuned.”*

Since the head’s weights still reflect the original pre-trained checkpoint rather than the fine-tuned model, any confidence value it produces would be statistically invalid. Rather than expose potentially misleading calibration, Needle nullifies the field entirely.

## How Needle Enforces This Safety Check

The library implements runtime detection of fine-tuned weights and actively suppresses confidence reporting. In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the code emits a warning on lines 161–162:

> *“finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None.”*

Subsequently, line 258 explicitly forces the confidence field to `None` when fine-tuned weights are detected. This ensures that every response from a fine-tuned model carries `response["confidence"] = None` regardless of the input prompt.

```python
from needle import Needle

# Pre-trained base model returns calibrated confidence

base_model = Needle(weights="base.cact")
base_resp = base_model("What is the capital of France?")
print(base_resp["confidence"])  # Output: 0.92 (example)

# Fine-tuned model returns None

ft_model = Needle(weights="my_finetuned.cact")
ft_resp = ft_model("What is the capital of France?")
print(ft_resp["confidence"])  # Output: None

```

## Practical Impact and Workarounds

### Detecting the Behavior in Code

Your application should explicitly check for `None` before relying on confidence scores. This pattern allows graceful degradation when working with fine-tuned checkpoints:

```python
response = model(user_query)
if response["confidence"] is None:
    # Handle confidence-agnostic logic

    processed_result = apply_fallback_heuristic(response)
else:
    # Use calibrated confidence for gating

    if response["confidence"] > 0.85:
        processed_result = auto_execute(response)

```

### Bypassing the Confidence Gate

Many testing environments allow you to disable confidence-based gating entirely. In `needle.environments.smart_home`, set `min_confidence` to `0.0` to process all responses regardless of calibration status:

```python
from needle.environments import smart_home

# Run evaluation without confidence filtering

smart_home.run_tests(min_confidence=0.0)

```

### Restoring Calibrated Confidence

To recover confidence scores after fine-tuning, you must **re-train or fine-tune the confidence head** independently. This typically involves:

1. Freezing the main transformer weights
2. Training only the `ConfidenceHead` module on a validation set
3. Replacing the head in your checkpoint via [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)

The confidence head identifiers (`contrastive_head`, `confidence_head`) are defined in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) and can be selectively updated without touching the main language model.

## Key Source Files

- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)**: Contains the architectural comment on line 443 explaining that the confidence head is excluded from the fine-tuning process.
- **[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)**: Implements the runtime warning (lines 161–162) and the logic that forces `response["confidence"] = None` (line 258) for fine-tuned weights.
- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)**: Defines the `ConfidenceHead` module structure responsible for converting hidden states into calibrated confidence values.
- **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)**: Lists the head identifiers used during checkpoint export and restoration, enabling selective updates of individual components.

## Summary

- **Fine-tuning updates only the transformer backbone**, leaving the confidence head with stale, pre-trained weights.
- **Uncalibrated confidence is dangerous**, so Needle returns `None` rather than misleading values.
- **Detection happens at runtime** in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), which emits warnings and nullifies the confidence field for fine-tuned checkpoints.
- **Workarounds exist**: Set `min_confidence=0.0` to bypass gating, or re-train the confidence head on validation data to restore calibrated scores.
- **Source files to inspect**: [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), and [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

## Frequently Asked Questions

### Why does Needle disable confidence after fine-tuning instead of keeping the old values?

The confidence head relies on specific statistical relationships between hidden states and prediction accuracy. When you fine-tune the main transformer, these relationships shift, rendering the original calibration curve invalid. Presenting stale confidence scores would give users false certainty about model predictions, potentially leading to incorrect automated decisions.

### Can I force Needle to show confidence scores even with fine-tuned weights?

While the library does not provide a direct override flag, you can bypass confidence-dependent gates by setting `min_confidence=0.0` in environment configurations. However, this merely ignores the missing confidence value; it does not generate a valid calibrated score. To actually obtain confidence values, you must re-train the head or use the original pre-trained weights.

### How do I recalibrate the confidence head after fine-tuning?

You need to run a secondary training phase where the main transformer weights are frozen and only the `ConfidenceHead` parameters are updated. Use a representative validation set to train the head to predict accuracy on your specific fine-tuned domain. Once trained, inject the new head weights into your checkpoint using the export utilities in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).

### Which model components are updated during Needle fine-tuning?

Only the **main transformer weights** (language model backbone) receive gradient updates during standard fine-tuning. Auxiliary heads—including the `contrastive_head` and `confidence_head` defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)—remain frozen at their pre-trained initialization values throughout the process.