Why Confidence Scores Are None When Using Fine-Tuned Weights in Needle 2

Confidence scores return None in Needle 2 when using fine-tuned weights because the confidence head is not updated during LoRA-style fine-tuning, rendering its outputs uncalibrated and potentially misleading.

When you instantiate the Needle engine with custom weights from a fine-tuned checkpoint, the library deliberately suppresses confidence scores to prevent downstream errors. According to the cactus-compute/needle source code, this behavior stems from the architectural separation between the trainable LoRA adapters and the frozen confidence calibration head.

The Architecture Behind Missing Confidence Scores

Why LoRA Adapters Skip the Confidence Head

Needle 2 generates confidence scores through a dedicated confidence head embedded within the base model architecture. This head resides in needle/model/architecture.py as the ConfidenceHead class and is responsible for calibrating prediction certainty.

During LoRA-style fine-tuning, the training process updates only the low-rank adapter layers while leaving the base model weights—and specifically the confidence head—frozen. Because the head never sees the new fine-tuning data, its calibration becomes invalid for the adapted model. Returning these miscalibrated values would break decision logic downstream, so the framework disables them entirely.

Engine Implementation: Where Scores Become None

Constructor Warnings in needle/__init__.py

The suppression logic begins in the constructor. When you instantiate the engine with the weights parameter (e.g., Needle(weights="my_finetuned.cact")), the code at lines 58-63 detects the custom checkpoint and emits a runtime warning: "finetuning does not update the confidence head, so scores are uncalibrated".

This check distinguishes between base model weights (bundled with the package) and user-provided fine-tuned checkpoints. The warning fires once during initialization to alert developers that certainty metrics will be unavailable.

The complete Method Override

After the constructor warns the user, the complete method enforces the None value at lines 21-23 of needle/__init__.py. After the engine generates the response envelope, the code explicitly checks for the presence of custom weights and overwrites any confidence field with None before returning the JSON object to the caller.

This guarantees that downstream applications never receive stale calibration values, regardless of what the underlying ConfidenceHead might compute.

Practical Examples: Base vs. Fine-Tuned Checkpoints

The following examples demonstrate the behavioral difference between base and fine-tuned deployments.

Using the base model returns valid confidence scores:

from needle import Needle

agent = Needle()                     # loads bundled base weights

resp = agent.complete("What is the capital of France?")
print(resp["confidence"])           # → 0.97 (calibrated score)

Loading fine-tuned weights triggers the suppression:

# Assume my_finetuned.cact was produced by needle finetune ...

agent_ft = Needle(weights="my_finetuned.cact")
resp_ft = agent_ft.complete("What is the capital of France?")
print(resp_ft["confidence"])        # → None

# Warning printed: "finetuning does not update the confidence head..."

For developers inspecting the architecture directly, the confidence head remains present but unused:

from needle.model.architecture import SimpleAttentionNetwork, ConfidenceHead

model = SimpleAttentionNetwork(config)
print(isinstance(model.confidence_head, ConfidenceHead))  # True

# After loading fine-tuned weights, the head exists but outputs are ignored

Documentation and Design Rationale

The design choice is documented explicitly in the project's markdown guides. The Finetuning guide at doc/finetuning.md (lines 75-84) states that the confidence head and its calibration remain unchanged when LoRA adapters merge into the checkpoint.

Similarly, the API documentation at doc/apis.md (lines 148-154) clarifies the contract: "Fine-tuning does not update the head, so an agent running tuned weights reports confidence as None."

This deliberate restriction ensures that production systems never act upon stale calibration data when using domain-specific fine-tuned models.

Summary

• Confidence scores require calibration by the ConfidenceHead class defined in needle/model/architecture.py.
• LoRA fine-tuning freezes the confidence head, leaving it uncalibrated for the adapted model weights.
• Needle 2 detects custom weight files in needle/__init__.py and emits a constructor warning at lines 58-63.
• The complete method forces None for the confidence field at lines 21-23 when fine-tuned weights are present.
• Documentation confirms this is intentional behavior to prevent misleading certainty scores in production.

Frequently Asked Questions

Can I manually re-enable confidence scores for fine-tuned weights?

No, the complete method in needle/__init__.py explicitly overwrites the confidence field with None at lines 21-23 whenever custom weights are detected. There is no configuration flag to bypass this safety mechanism, as the underlying ConfidenceHead lacks calibration for the fine-tuned adapter parameters.

Does this limitation apply to all fine-tuning methods in Needle 2?

The analysis specifically addresses LoRA-style fine-tuning, which is the primary method documented in doc/finetuning.md. The confidence head remains frozen during this process because LoRA adapters target specific attention layers while excluding the calibration head from the trainable parameter set.

How can I obtain confidence scores after fine-tuning my model?

Currently, Needle 2 does not support recalibrating the confidence head post-fine-tuning. To receive valid confidence scores, you must use the base model weights without custom fine-tuned checkpoints. The framework treats the base checkpoint as the only source of calibrated certainty metrics.

Is the confidence head removed from the model architecture when fine-tuning?

No, the ConfidenceHead remains instantiated within the SimpleAttentionNetwork class and exists in memory. However, the engine ignores its outputs and returns None instead, effectively disabling the feature without altering the model architecture or removing the module from the computational graph.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}