Confidence Scores for Fine-Tuned Weights in Needle: Calibration Note Explained

Needle disables confidence scores entirely when using fine-tuned weights because the confidence head is calibrated only for the base pre-trained model and remains unchanged during LoRA fine-tuning.

Needle is an open-source agent framework that provides calibrated confidence scores to help developers assess prediction reliability. However, this calibration has a critical limitation when working with custom fine-tuned models. Understanding this behavior prevents deployment issues and ensures you interpret model outputs correctly.

Why Fine-Tuned Weights Break Confidence Calibration

The ConfidenceHead defined in needle/model/architecture.py (lines 488-492) is trained and calibrated exclusively on the base model's distribution. When you fine-tune a model using LoRA adapters, this component receives no gradient updates and retains its original calibration.

This creates a fundamental mismatch: the fine-tuned model's predictions shift, but the confidence scoring mechanism does not adapt accordingly. Un calibrated confidence scores would be misleading, potentially causing over-reliiance on incorrect predictions.

How Needle Handles This at Runtime

At instantiation, Needle(weights=...) performs automatic detection of tuned weights. According to the implementation in needle/__init__.py (lines 161-162), the package:

1. Emits a one-time warning alerting you that confidence is unavailable
2. Sets all confidence values to None in response objects
3. Suppresses confidence head computation entirely for efficiency

This design prioritizes explicit failure over silent degradation— you immediately know when confidence is unreliable.

Practical Code Examples

Base Model: Confidence Available

import needle

base_agent = needle.Needle(
    tools=[weather_lookup, calendar_check],
    weights="base.cact"
)

response = base_agent.run("What's the weather in Tokyo?")
print(response["confidence"])

# → 0.93  # Calibrated confidence score

Fine-Tuned Model: Confidence Disabled

import needle

tuned_agent = needle.Needle(
    tools=[weather_lookup, calendar_check],
    weights="tuned.cact"  # LoRA fine-tuned weights

)

response = tuned_agent.run("What's the weather in Tokyo?")
print(response.get("confidence"))

# → None

# Console warning on first instantiation:

# "finetuning does not update the confidence head, so scores are

#  uncalibrated for tuned weights; this agent reports confidence as None"

Documentation References

The Needle documentation explicitly covers this limitation in two key locations:

• doc/finetuning.md (lines 85-88): Explains under "What finetuning does not change → The confidence head" that the confidence head is untouched during LoRA fine-tuning and that the package disables confidence for tuned weights

• doc/apis.md (lines 68-73): Details in the Confidence paragraph that confidence calculation has calibration limits and describes the effect of fine-tuning on the confidence field

What This Means for Production Deployments

When deploying a fine-tuned Needle model, you should:

• Remove any logic dependent on confidence thresholds from your application
• Implement alternative quality assurance (human-in-the-loop, secondary verification, or output consistency checks)
• Document this limitation for downstream consumers of your API
• Consider re-calibration only if you have substantial validation data and can train a custom confidence head

Summary

• Confidence scores are base-model only: The ConfidenceHead in needle/model/architecture.py never updates during LoRA fine-tuning
• Automatic disablement: needle/__init__.py detects tuned weights and sets confidence: None with a runtime warning
• Documentation verified: Both doc/finetuning.md and doc/apis.md explicitly document this behavior
• No workaround available: You must use base weights if confidence scores are required for your use case

Frequently Asked Questions

Can I manually enable confidence scores for fine-tuned weights?

No. The Needle package explicitly prevents this because uncalibrated confidence scores would be actively harmful—overconfident wrong predictions are worse than acknowledged uncertainty. There is no configuration flag to override this safety mechanism.

Will future versions support confidence calibration for fine-tuned models?

The current architecture treats the confidence head as frozen infrastructure. Adding calibration support for fine-tuned weights would require either fine-tuning the confidence head itself (with appropriate validation data) or implementing domain adaptation techniques, neither of which is on the current roadmap according to the doc/finetuning.md documentation.

How can I assess prediction quality without confidence scores?

Several alternatives exist: implement consistency checks by running the same query with minor variations, use token-level surprisal from the underlying language model, add a secondary verification step with a different model, or establish human review thresholds based on query complexity or domain criticality.

Does this limitation apply to all types of fine-tuning?

Currently, yes. The documentation specifically mentions LoRA adapters, but the underlying mechanism—confidence head immutability—would affect any fine-tuning approach that does not explicitly retrain the confidence head component defined in needle/model/architecture.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}