# How Tuned Weights in Needle 2 Disable Confidence Scoring

> Discover why tuned weights in Needle 2 disable confidence scoring. Learn how fine-tuning impacts the confidence head and what happens to your results.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-21

---

**When you load tuned weights in Needle 2, the confidence field is explicitly set to `None` because the fine-tuning process updates only the language and tool-use components while leaving the confidence head untrained.**

Needle 2 is an open-source agent framework that normally returns calibrated confidence scores with each completion, but this behavior changes when using fine-tuned checkpoints. When you load a model produced by the `needle finetune` workflow, the agent intentionally reports `None` for confidence values rather than uncalibrated numeric scores. This implementation detail is hardcoded in the initialization logic to prevent misleading confidence outputs from outdated calibration parameters.

## The Confidence Head Architecture

The confidence scoring mechanism in Needle 2 relies on a dedicated **confidence head** defined separately from the main language model. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 63-76), the `ConfidenceHead` module processes hidden states to produce probability estimates. However, this component is architecturally isolated from the layers updated during fine-tuning.

Because the confidence head uses frozen parameters that are not affected by LoRA adapters or other weight-tuning steps, any checkpoint created through the `needle finetune` command retains the base model's confidence calibration. Since this calibration becomes invalid for fine-tuned behavior, the framework disables confidence reporting entirely.

## Initialization Warnings and Runtime Behavior

When you instantiate a Needle 2 agent with tuned weights, the constructor emits a specific warning during initialization. According to the source code in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 59-62), the framework logs:

> "finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None"

After each completion, the response object is programmatically patched to enforce this policy. In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 123-125), the code explicitly sets the `confidence` field to `None` whenever tuned weights are detected, ensuring that downstream applications cannot access stale numeric values.

## Comparing Base and Tuned Weight Behavior

The following examples demonstrate the behavioral difference between base model inference and fine-tuned inference.

### Base Model Returns Numeric Confidence

When using the default base model without custom weights, Needle 2 returns a floating-point confidence score:

```python
from needle import Needle

agent = Needle()                     # No weights → uses the default base model

result = agent.complete("What is the capital of France?")
print(result["confidence"])          # → e.g., 0.92

```

### Tuned Weights Return None

When loading a checkpoint produced by `needle finetune`, the confidence field becomes `None`:

```python
from needle import Needle

# `my_finetuned.cact` is a checkpoint produced by `needle finetune …`

agent = Needle(weights="my_finetuned.cact")
result = agent.complete("What is the capital of France?")
print(result["confidence"])          # → None

```

### Warning Output

The initialization warning appears when constructing the agent:

```python
import warnings
warnings.filterwarnings("ignore")   # hide the warning for clean output

agent = Needle(weights="my_finetuned.cact")

# Console output includes:

# finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None

```

## Key Implementation Files

Several source files control this behavior across the Needle 2 codebase:

- **[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)** – Contains the agent initialization logic that emits warnings (lines 59-62) and patches response objects to set `confidence = None` (lines 123-125).

- **[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)** – Defines the `ConfidenceHead` class (lines 63-76) that remains frozen during fine-tuning.

- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)** – Prints calibration notes during the training process and confirms that confidence outputs will be disabled for the resulting checkpoint.

- **[`tests/test_weights.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_weights.py)** – Contains test fixtures that verify confidence field presence for base models and absence for tuned weights.

## Summary

- **Tuned weights disable confidence**: Needle 2 explicitly returns `None` for confidence scores when loading fine-tuned checkpoints to prevent uncalibrated outputs.
- **Confidence head is frozen**: The architecture separates confidence estimation from language modeling, and fine-tuning only updates the latter components in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- **Warnings at initialization**: The framework logs explicit notifications in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) when detecting tuned weights to alert developers.
- **Runtime enforcement**: Response objects are patched after each completion to ensure the confidence field remains `None` regardless of base model defaults.

## Frequently Asked Questions

### Why does Needle 2 return None for confidence with tuned weights?

Needle 2 returns `None` because the confidence head defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) is not updated during the fine-tuning process. Since the calibration would be invalid for the new weight distribution, the framework in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) deliberately disables reporting to prevent misleading confidence values.

### Can I calibrate the confidence head after fine-tuning?

The current implementation in the cactus-compute/needle repository does not provide a mechanism to recalibrate the confidence head after fine-tuning. The `ConfidenceHead` module remains frozen with its base model parameters, and the `needle finetune` workflow does not include calibration steps for this component.

### Does the base model provide reliable confidence scores?

Yes, when using the base model without tuned weights, Needle 2 returns numeric confidence scores (typically between 0.0 and 1.0) from the fully calibrated confidence head. These values are only disabled when loading checkpoints that have undergone the fine-tuning process.

### Which source files control confidence behavior in Needle 2?

The primary files are [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (which emits warnings and forces `None` values), [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (which defines the frozen `ConfidenceHead`), and [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) (which documents the limitation during training). Together, these components ensure that tuned weights operate without uncalibrated confidence outputs.