# Why Confidence Scores Are None When Using Fine-Tuned Weights in Needle 2

> Discover why confidence scores are None in Needle 2 with fine-tuned weights. Learn how LoRA fine-tuning affects the confidence head and get insights to improve your model's reliability.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-19

---

**Confidence scores return `None` in Needle 2 when using fine-tuned weights because the confidence head is not updated during LoRA-style fine-tuning, rendering its outputs uncalibrated and potentially misleading.**

When you instantiate the `Needle` engine with custom weights from a fine-tuned checkpoint, the library deliberately suppresses confidence scores to prevent downstream errors. According to the `cactus-compute/needle` source code, this behavior stems from the architectural separation between the trainable LoRA adapters and the frozen confidence calibration head.

## The Architecture Behind Missing Confidence Scores

### Why LoRA Adapters Skip the Confidence Head

Needle 2 generates confidence scores through a dedicated **confidence head** embedded within the base model architecture. This head resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) as the `ConfidenceHead` class and is responsible for calibrating prediction certainty.

During LoRA-style fine-tuning, the training process updates only the low-rank adapter layers while leaving the base model weights—and specifically the confidence head—frozen. Because the head never sees the new fine-tuning data, its calibration becomes invalid for the adapted model. Returning these miscalibrated values would break decision logic downstream, so the framework disables them entirely.

## Engine Implementation: Where Scores Become None

### Constructor Warnings in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)

The suppression logic begins in the constructor. When you instantiate the engine with the `weights` parameter (e.g., `Needle(weights="my_finetuned.cact")`), the code at lines 58-63 detects the custom checkpoint and emits a runtime warning: *"finetuning does not update the confidence head, so scores are uncalibrated"*.

This check distinguishes between base model weights (bundled with the package) and user-provided fine-tuned checkpoints. The warning fires once during initialization to alert developers that certainty metrics will be unavailable.

### The `complete` Method Override

After the constructor warns the user, the `complete` method enforces the `None` value at lines 21-23 of [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py). After the engine generates the response envelope, the code explicitly checks for the presence of custom weights and overwrites any confidence field with `None` before returning the JSON object to the caller.

This guarantees that downstream applications never receive stale calibration values, regardless of what the underlying `ConfidenceHead` might compute.

## Practical Examples: Base vs. Fine-Tuned Checkpoints

The following examples demonstrate the behavioral difference between base and fine-tuned deployments.

Using the base model returns valid confidence scores:

```python
from needle import Needle

agent = Needle()                     # loads bundled base weights

resp = agent.complete("What is the capital of France?")
print(resp["confidence"])           # → 0.97 (calibrated score)

```

Loading fine-tuned weights triggers the suppression:

```python

# Assume my_finetuned.cact was produced by needle finetune ...

agent_ft = Needle(weights="my_finetuned.cact")
resp_ft = agent_ft.complete("What is the capital of France?")
print(resp_ft["confidence"])        # → None

# Warning printed: "finetuning does not update the confidence head..."

```

For developers inspecting the architecture directly, the confidence head remains present but unused:

```python
from needle.model.architecture import SimpleAttentionNetwork, ConfidenceHead

model = SimpleAttentionNetwork(config)
print(isinstance(model.confidence_head, ConfidenceHead))  # True

# After loading fine-tuned weights, the head exists but outputs are ignored

```

## Documentation and Design Rationale

The design choice is documented explicitly in the project's markdown guides. The **Finetuning** guide at [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) (lines 75-84) states that the confidence head and its calibration remain unchanged when LoRA adapters merge into the checkpoint.

Similarly, the **API** documentation at [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) (lines 148-154) clarifies the contract: *"Fine-tuning does not update the head, so an agent running tuned weights reports `confidence` as `None`."*

This deliberate restriction ensures that production systems never act upon stale calibration data when using domain-specific fine-tuned models.

## Summary

- **Confidence scores require calibration** by the `ConfidenceHead` class defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- **LoRA fine-tuning freezes the confidence head**, leaving it uncalibrated for the adapted model weights.
- **Needle 2 detects custom weight files** in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) and emits a constructor warning at lines 58-63.
- **The `complete` method forces `None`** for the confidence field at lines 21-23 when fine-tuned weights are present.
- **Documentation confirms** this is intentional behavior to prevent misleading certainty scores in production.

## Frequently Asked Questions

### Can I manually re-enable confidence scores for fine-tuned weights?

No, the `complete` method in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) explicitly overwrites the confidence field with `None` at lines 21-23 whenever custom weights are detected. There is no configuration flag to bypass this safety mechanism, as the underlying `ConfidenceHead` lacks calibration for the fine-tuned adapter parameters.

### Does this limitation apply to all fine-tuning methods in Needle 2?

The analysis specifically addresses LoRA-style fine-tuning, which is the primary method documented in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md). The confidence head remains frozen during this process because LoRA adapters target specific attention layers while excluding the calibration head from the trainable parameter set.

### How can I obtain confidence scores after fine-tuning my model?

Currently, Needle 2 does not support recalibrating the confidence head post-fine-tuning. To receive valid confidence scores, you must use the base model weights without custom fine-tuned checkpoints. The framework treats the base checkpoint as the only source of calibrated certainty metrics.

### Is the confidence head removed from the model architecture when fine-tuning?

No, the `ConfidenceHead` remains instantiated within the `SimpleAttentionNetwork` class and exists in memory. However, the engine ignores its outputs and returns `None` instead, effectively disabling the feature without altering the model architecture or removing the module from the computational graph.