Why Confidence Reports None After Fine-Tuning a Needle Model

Needle deliberately sets response["confidence"] to None after fine-tuning because the confidence head—a separate calibration network—is not updated during the fine-tuning process, leaving it uncalibrated for the new weights.

When working with the cactus-compute/needle repository, developers often encounter a sudden loss of confidence scores after fine-tuning a model. This behavior is intentional and protects against misleading predictions. The library intentionally withholds confidence values when it detects that a model has been fine-tuned to prevent uncalibrated outputs from affecting downstream decision-making.

The Root Cause: Untouched Confidence Head

Fine-tuning in Needle updates only the main transformer weights (the language model backbone) while leaving the confidence head completely unchanged. The confidence head is a small auxiliary network that converts hidden-state statistics into calibrated probability scores. Because this head remains frozen at its pre-trained weights during fine-tuning, it becomes misaligned with the newly tuned language model.

In needle/model/finetune.py, line 443 contains a comment documenting this design decision:

“confidence reports None with tuned weights; the head is not tuned.”

Since the head’s weights still reflect the original pre-trained checkpoint rather than the fine-tuned model, any confidence value it produces would be statistically invalid. Rather than expose potentially misleading calibration, Needle nullifies the field entirely.

How Needle Enforces This Safety Check

The library implements runtime detection of fine-tuned weights and actively suppresses confidence reporting. In needle/__init__.py, the code emits a warning on lines 161–162:

“finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None.”

Subsequently, line 258 explicitly forces the confidence field to None when fine-tuned weights are detected. This ensures that every response from a fine-tuned model carries response["confidence"] = None regardless of the input prompt.

from needle import Needle

# Pre-trained base model returns calibrated confidence

base_model = Needle(weights="base.cact")
base_resp = base_model("What is the capital of France?")
print(base_resp["confidence"])  # Output: 0.92 (example)

# Fine-tuned model returns None

ft_model = Needle(weights="my_finetuned.cact")
ft_resp = ft_model("What is the capital of France?")
print(ft_resp["confidence"])  # Output: None

Practical Impact and Workarounds

Detecting the Behavior in Code

Your application should explicitly check for None before relying on confidence scores. This pattern allows graceful degradation when working with fine-tuned checkpoints:

response = model(user_query)
if response["confidence"] is None:
    # Handle confidence-agnostic logic

    processed_result = apply_fallback_heuristic(response)
else:
    # Use calibrated confidence for gating

    if response["confidence"] > 0.85:
        processed_result = auto_execute(response)

Bypassing the Confidence Gate

Many testing environments allow you to disable confidence-based gating entirely. In needle.environments.smart_home, set min_confidence to 0.0 to process all responses regardless of calibration status:

from needle.environments import smart_home

# Run evaluation without confidence filtering

smart_home.run_tests(min_confidence=0.0)

Restoring Calibrated Confidence

To recover confidence scores after fine-tuning, you must re-train or fine-tune the confidence head independently. This typically involves:

1. Freezing the main transformer weights
2. Training only the ConfidenceHead module on a validation set
3. Replacing the head in your checkpoint via needle/model/export.py

The confidence head identifiers (contrastive_head, confidence_head) are defined in needle/model/export.py and can be selectively updated without touching the main language model.

Key Source Files

• needle/model/finetune.py: Contains the architectural comment on line 443 explaining that the confidence head is excluded from the fine-tuning process.
• needle/__init__.py: Implements the runtime warning (lines 161–162) and the logic that forces response["confidence"] = None (line 258) for fine-tuned weights.
• needle/model/architecture.py: Defines the ConfidenceHead module structure responsible for converting hidden states into calibrated confidence values.
• needle/model/export.py: Lists the head identifiers used during checkpoint export and restoration, enabling selective updates of individual components.

Summary

• Fine-tuning updates only the transformer backbone, leaving the confidence head with stale, pre-trained weights.
• Uncalibrated confidence is dangerous, so Needle returns None rather than misleading values.
• Detection happens at runtime in needle/__init__.py, which emits warnings and nullifies the confidence field for fine-tuned checkpoints.
• Workarounds exist: Set min_confidence=0.0 to bypass gating, or re-train the confidence head on validation data to restore calibrated scores.
• Source files to inspect: needle/model/finetune.py, needle/__init__.py, and needle/model/architecture.py.

Frequently Asked Questions

Why does Needle disable confidence after fine-tuning instead of keeping the old values?

The confidence head relies on specific statistical relationships between hidden states and prediction accuracy. When you fine-tune the main transformer, these relationships shift, rendering the original calibration curve invalid. Presenting stale confidence scores would give users false certainty about model predictions, potentially leading to incorrect automated decisions.

Can I force Needle to show confidence scores even with fine-tuned weights?

While the library does not provide a direct override flag, you can bypass confidence-dependent gates by setting min_confidence=0.0 in environment configurations. However, this merely ignores the missing confidence value; it does not generate a valid calibrated score. To actually obtain confidence values, you must re-train the head or use the original pre-trained weights.

How do I recalibrate the confidence head after fine-tuning?

You need to run a secondary training phase where the main transformer weights are frozen and only the ConfidenceHead parameters are updated. Use a representative validation set to train the head to predict accuracy on your specific fine-tuned domain. Once trained, inject the new head weights into your checkpoint using the export utilities in needle/model/export.py.

Which model components are updated during Needle fine-tuning?

Only the main transformer weights (language model backbone) receive gradient updates during standard fine-tuning. Auxiliary heads—including the contrastive_head and confidence_head defined in needle/model/architecture.py—remain frozen at their pre-trained initialization values throughout the process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}