Why Does Confidence Return None After Fine-Tuning in Needle: Causes and Solutions

Needle deliberately returns confidence: None after LoRA fine-tuning because the ConfidenceHead module remains frozen during adapter training, leaving scores uncalibrated and unreliable.

The needle library from cactus-compute provides tool-selection capabilities with built-in confidence scoring, but confidence returns None after fine-tuning by design. This behavior protects your application from misleading metrics, as the LoRA training process updates only the contrastive head while leaving the ConfidenceHead untouched, ensuring uncalibrated scores are never exposed to production code.

Why Confidence Returns None After Fine-Tuning in Needle

The Needle class relies on two distinct neural network heads defined in needle/model/architecture.py: the ContrastiveHead for tool selection and the ConfidenceHead for scoring prediction reliability. According to the source code in needle/model/architecture.py (lines 63-76), the ConfidenceHead pools learned "probe" embeddings and projects them to a single logit interpreted as the confidence score.

During LoRA fine-tuning, implemented in needle/model/finetune.py, only the contrastive head parameters are updated to adapt tool selection behavior. The ConfidenceHead remains frozen at its base checkpoint values. Since the confidence parameters are not adjusted to reflect the new fine-tuned distribution, the scores become statistically uncalibrated. The test suite in tests/test_finetune.py validates this behavior and the subsequent None handling.

How Needle Prevents Uncalibrated Confidence Exposure

The library implements safeguards in needle/__init__.py to prevent misleading confidence values from reaching your application.

When you load a custom fine-tuned checkpoint, the constructor emits a warning (lines 58-60) indicating that "finetuning does not update the confidence head, so scores are uncalibrated..." This alerts developers that confidence scores would be unreliable.

Subsequently, in the complete method (lines 15-17), the runtime explicitly overwrites any confidence value with None when custom weights are detected. This hardcoded safety mechanism ensures that downstream applications cannot accidentally depend on stale confidence metrics from the frozen ConfidenceHead.

How to Handle Missing Confidence in Fine-Tuned Models

When working with fine-tuned Needle checkpoints, you have several strategies to manage the missing confidence field.

Use the Base Model for Calibrated Scores

If your workflow requires confidence scores, instantiate the base model rather than the fine-tuned adapter:

from needle import Needle

# Load base model to get calibrated confidence

agent = Needle(weights="base_model.cact", tools=[...])
response = agent.complete("Summarize the article")
print(response["confidence"])  # Output: 0.87 (float between 0 and 1)

Accept None for Fine-Tuned Workflows

Design your application to handle None confidence when using fine-tuned weights:

# Fine-tuned checkpoint lacks confidence calibration

agent = Needle(weights="my_finetuned.cact", tools=[...])
response = agent.complete("What is the weather today?")
print(response["confidence"])  # Output: None

if response["confidence"] is None:
    # Implement fallback logic or skip confidence-based filtering

    pass

Understanding the Default Fine-Tuning Behavior

The standard fine-tuning command intentionally skips confidence head training to reduce memory and compute requirements:

needle finetune \
    --checkpoint base_model.cact \
    --lora-rank 8 \
    --out my_finetuned.cact \
    --epochs 3

This command, referenced in needle/model/finetune.py (line 401), produces a LoRA adapter that modifies only the contrastive head, which is why the runtime reports None for confidence after loading my_finetuned.cact.

Summary

• The ConfidenceHead in needle/model/architecture.py (lines 63-76) remains frozen during LoRA fine-tuning, while only the ContrastiveHead receives gradient updates.
• Needle deliberately returns confidence: None after fine-tuning to prevent uncalibrated scores from misleading your application.
• The complete method in needle/__init__.py (lines 15-17) enforces this safety check by overwriting confidence values when custom weights are detected.
• Use the base model checkpoint if your workflow requires calibrated confidence scores, or implement None handling for fine-tuned deployments.

Frequently Asked Questions

Why is confidence None only after fine-tuning?

Confidence returns None exclusively after fine-tuning because the LoRA training process updates only the contrastive head parameters used for tool selection, leaving the ConfidenceHead at its base checkpoint values. The complete method detects the presence of custom weights and forces the confidence field to None to prevent uncalibrated outputs from reaching your code.

Can I train the confidence head during fine-tuning?

The current implementation in needle/model/finetune.py does not support updating the ConfidenceHead during LoRA fine-tuning. The training loop intentionally freezes these parameters to reduce computational overhead and memory usage. Modifying the source code to unfreeze the ConfidenceHead would require substantial changes to the fine-tuning script and retraining from scratch.

How do I get confidence scores with a fine-tuned model?

To obtain confidence scores, instantiate Needle with the base model weights rather than the fine-tuned adapter file. The base model contains calibrated ConfidenceHead parameters that produce reliable scores between 0 and 1. You cannot generate valid confidence scores from fine-tuned checkpoints without modifying the training architecture to update the ConfidenceHead.

Is the confidence score reliable in the base model?

Yes, confidence scores are reliable when using the base (non-fine-tuned) model because the ConfidenceHead parameters remain synchronized with the base checkpoint. These scores represent the model's calibrated self-assessment of tool selection accuracy and are suitable for threshold-based filtering or uncertainty quantification in production environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}