How Tuned Weights in Needle 2 Disable Confidence Scoring

When you load tuned weights in Needle 2, the confidence field is explicitly set to None because the fine-tuning process updates only the language and tool-use components while leaving the confidence head untrained.

Needle 2 is an open-source agent framework that normally returns calibrated confidence scores with each completion, but this behavior changes when using fine-tuned checkpoints. When you load a model produced by the needle finetune workflow, the agent intentionally reports None for confidence values rather than uncalibrated numeric scores. This implementation detail is hardcoded in the initialization logic to prevent misleading confidence outputs from outdated calibration parameters.

The Confidence Head Architecture

The confidence scoring mechanism in Needle 2 relies on a dedicated confidence head defined separately from the main language model. In needle/model/architecture.py (lines 63-76), the ConfidenceHead module processes hidden states to produce probability estimates. However, this component is architecturally isolated from the layers updated during fine-tuning.

Because the confidence head uses frozen parameters that are not affected by LoRA adapters or other weight-tuning steps, any checkpoint created through the needle finetune command retains the base model's confidence calibration. Since this calibration becomes invalid for fine-tuned behavior, the framework disables confidence reporting entirely.

Initialization Warnings and Runtime Behavior

When you instantiate a Needle 2 agent with tuned weights, the constructor emits a specific warning during initialization. According to the source code in needle/__init__.py (lines 59-62), the framework logs:

"finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None"

After each completion, the response object is programmatically patched to enforce this policy. In needle/__init__.py (lines 123-125), the code explicitly sets the confidence field to None whenever tuned weights are detected, ensuring that downstream applications cannot access stale numeric values.

Comparing Base and Tuned Weight Behavior

The following examples demonstrate the behavioral difference between base model inference and fine-tuned inference.

Base Model Returns Numeric Confidence

When using the default base model without custom weights, Needle 2 returns a floating-point confidence score:

from needle import Needle

agent = Needle()                     # No weights → uses the default base model

result = agent.complete("What is the capital of France?")
print(result["confidence"])          # → e.g., 0.92

Tuned Weights Return None

When loading a checkpoint produced by needle finetune, the confidence field becomes None:

from needle import Needle

# `my_finetuned.cact` is a checkpoint produced by `needle finetune …`

agent = Needle(weights="my_finetuned.cact")
result = agent.complete("What is the capital of France?")
print(result["confidence"])          # → None

Warning Output

The initialization warning appears when constructing the agent:

import warnings
warnings.filterwarnings("ignore")   # hide the warning for clean output

agent = Needle(weights="my_finetuned.cact")

# Console output includes:

# finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None

Key Implementation Files

Several source files control this behavior across the Needle 2 codebase:

• needle/__init__.py – Contains the agent initialization logic that emits warnings (lines 59-62) and patches response objects to set confidence = None (lines 123-125).

• needle/model/architecture.py – Defines the ConfidenceHead class (lines 63-76) that remains frozen during fine-tuning.

• needle/model/finetune.py – Prints calibration notes during the training process and confirms that confidence outputs will be disabled for the resulting checkpoint.

• tests/test_weights.py – Contains test fixtures that verify confidence field presence for base models and absence for tuned weights.

Summary

• Tuned weights disable confidence: Needle 2 explicitly returns None for confidence scores when loading fine-tuned checkpoints to prevent uncalibrated outputs.
• Confidence head is frozen: The architecture separates confidence estimation from language modeling, and fine-tuning only updates the latter components in needle/model/architecture.py.
• Warnings at initialization: The framework logs explicit notifications in needle/__init__.py when detecting tuned weights to alert developers.
• Runtime enforcement: Response objects are patched after each completion to ensure the confidence field remains None regardless of base model defaults.

Frequently Asked Questions

Why does Needle 2 return None for confidence with tuned weights?

Needle 2 returns None because the confidence head defined in needle/model/architecture.py is not updated during the fine-tuning process. Since the calibration would be invalid for the new weight distribution, the framework in needle/__init__.py deliberately disables reporting to prevent misleading confidence values.

Can I calibrate the confidence head after fine-tuning?

The current implementation in the cactus-compute/needle repository does not provide a mechanism to recalibrate the confidence head after fine-tuning. The ConfidenceHead module remains frozen with its base model parameters, and the needle finetune workflow does not include calibration steps for this component.

Does the base model provide reliable confidence scores?

Yes, when using the base model without tuned weights, Needle 2 returns numeric confidence scores (typically between 0.0 and 1.0) from the fully calibrated confidence head. These values are only disabled when loading checkpoints that have undergone the fine-tuning process.

Which source files control confidence behavior in Needle 2?

The primary files are needle/__init__.py (which emits warnings and forces None values), needle/model/architecture.py (which defines the frozen ConfidenceHead), and needle/model/finetune.py (which documents the limitation during training). Together, these components ensure that tuned weights operate without uncalibrated confidence outputs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}