Why Confidence Reports None with Fine-Tuned Weights in Needle and How to Handle It

confidence returns None for fine-tuned Needle models because the calibration head is not updated during fine-tuning, making any score potentially misleading.

When you load a fine-tuned checkpoint in Needle—such as one with LoRA adapters—the confidence head remains frozen while only the language model weights change. This design protects you from over- or under-confident predictions, but it requires understanding how to work around the missing scores when you need them.

Why Fine-Tuned Weights Disable Confidence Scoring

The ConfidenceHead in Needle is trained separately on the base model's output distribution. Fine-tuning with techniques like LoRA modifies that distribution without touching the head, breaking the calibration. Returning a number would be worse than returning nothing.

In needle/__init__.py, the constructor detects user-supplied weights and explicitly forces response["confidence"] = None:

# From needle/__init__.py#L58-L60

if self.has_custom_weights:
    import warnings
    warnings.warn("Confidence scores are not available with fine-tuned weights; "
                  "the confidence head remains uncalibrated.")
    self._confidence_available = False

The same limitation is documented in needle/model/finetune.py#L99-L102 where the CLI notes: "confidence reports None with tuned weights; the head is not tuned."

How to Handle Missing Confidence Scores

Choose your approach based on whether you need probability estimates:

Option 1: Ignore Confidence If Not Required

For deterministic tasks—classification, extraction, or tool calling where you only need the answer—simply omit confidence from your logic:

from needle import Needle

tuned = Needle(weights="model_lora.cact")
response = tuned.complete("Extract the date from this invoice.")
print(response["text"])  # Valid output

# response["confidence"] is None — safe to ignore

Option 2: Use Base Weights for Calibrated Confidence

Run inference twice: once with fine-tuned weights for quality, once with base weights for the score:

base = Needle(weights="model.cact")
tuned = Needle(weights="model_lora.cact")

def complete_with_confidence(prompt):
    # Get high-quality response from tuned model

    tuned_resp = tuned.complete(prompt)
    
    # If confidence missing, re-query base model

    if tuned_resp.get("confidence") is None:
        base_resp = base.complete(prompt)
        tuned_resp["confidence"] = base_resp.get("confidence")
    
    return tuned_resp

result = complete_with_confidence("Summarize this legal document.")
print(f"Answer: {result['text'][:100]}...")
print(f"Confidence: {result['confidence']:.2f}")

Option 3: Implement a Fallback Heuristic

Substitute None with a custom estimate based on available signals:

def estimate_confidence(response, model=None):
    """Fallback confidence estimation when calibration is unavailable."""
    if response.get("confidence") is not None:
        return response["confidence"]
    
    # Heuristic: use response length as proxy (shorter = more confident)

    text = response.get("text", "")
    token_count = len(text.split())
    return max(0.0, 1.0 - (token_count / 100))  # Simple inverse scaling

# Usage

response = tuned.complete("Translate to French: Hello world.")
print(f"Estimated confidence: {estimate_confidence(response):.2f}")

Option 4: Suppress Warnings in Test Suites

Capture and silence the calibration warning when testing:

import warnings
from needle import Needle

def test_finetuned_model():
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        model = Needle(weights="model_lora.cact")
        # ... test assertions ...

Architecture Reference: Where the Logic Lives

File	Purpose
needle/__init__.py	Constructor warning and None enforcement for custom weights
needle/model/finetune.py	CLI documentation of the limitation post-LoRA build
needle/model/architecture.py	ConfidenceHead definition (lines 488–576); unchanged during fine-tuning

The ConfidenceHead remains at its pre-training state because fine-tuning calls in finetune.py only update adapter layers, never touching the separate calibration network.

Re-Calibrating the Confidence Head (Advanced)

Currently, re-training the confidence head is not exposed in the public API. The head would need training data from the fine-tuned distribution to produce valid scores. Until this becomes available, the recommended paths are: use base weights, apply heuristics, or accept the absence of calibrated uncertainty.

Summary

• Root cause: The ConfidenceHead is not updated during fine-tuning, so confidence returns None to prevent misleading scores.
• Detection: Check needle/__init__.py constructor logic that warns and sets _confidence_available = False.
• Workarounds: Ignore the field, query base weights separately, implement heuristics, or suppress warnings.
• No API yet: Re-calibration requires internal training not currently exposed.

Frequently Asked Questions

What triggers the None confidence value in Needle?

Loading any checkpoint with modified weights—LoRA adapters, continued pre-training, or full fine-tuning—disables confidence. The constructor in needle/__init__.py checks has_custom_weights and forces the field to None regardless of fine-tuning method.

Can I enable confidence scoring for my fine-tuned model?

Not through the public API. The ConfidenceHead in needle/model/architecture.py requires re-training on outputs from your specific fine-tuned checkpoint. Until Cactus Compute exposes this functionality, you must use base weights or custom heuristics.

Does the None confidence affect other response fields?

No. response["text"], tool calls, and all other outputs remain fully valid. Only the calibrated probability estimate is withheld. Your application logic should treat None as "uncertainty unknown" rather than failure.

How can I silence the calibration warning in production?

Wrap instantiation in warnings.catch_warnings() as shown in Option 4 above. In needle/__init__.py, the warning is emitted via the standard library, so standard Python warning filters apply.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}