# Confidence Scores for Fine-Tuned Weights in Needle: Calibration Note Explained

> Learn why Needle disables confidence scores for fine-tuned weights. Understand the calibration note and its impact on LoRA models. Get clear insights for your ML projects.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-06

---

**Needle disables confidence scores entirely when using fine-tuned weights because the confidence head is calibrated only for the base pre-trained model and remains unchanged during LoRA fine-tuning.**

Needle is an open-source agent framework that provides calibrated confidence scores to help developers assess prediction reliability. However, this calibration has a critical limitation when working with custom fine-tuned models. Understanding this behavior prevents deployment issues and ensures you interpret model outputs correctly.

## Why Fine-Tuned Weights Break Confidence Calibration

The **`ConfidenceHead`** defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 488-492) is trained and calibrated exclusively on the base model's distribution. When you fine-tune a model using LoRA adapters, this component receives **no gradient updates** and retains its original calibration.

This creates a fundamental mismatch: the fine-tuned model's predictions shift, but the confidence scoring mechanism does not adapt accordingly. Un calibrated confidence scores would be misleading, potentially causing over-reliiance on incorrect predictions.

## How Needle Handles This at Runtime

At instantiation, `Needle(weights=...)` performs automatic detection of tuned weights. According to the implementation in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 161-162), the package:

1. **Emits a one-time warning** alerting you that confidence is unavailable
2. **Sets all confidence values to `None`** in response objects
3. **Suppresses confidence head computation** entirely for efficiency

This design prioritizes explicit failure over silent degradation— you immediately know when confidence is unreliable.

## Practical Code Examples

### Base Model: Confidence Available

```python
import needle

base_agent = needle.Needle(
    tools=[weather_lookup, calendar_check],
    weights="base.cact"
)

response = base_agent.run("What's the weather in Tokyo?")
print(response["confidence"])

# → 0.93  # Calibrated confidence score

```

### Fine-Tuned Model: Confidence Disabled

```python
import needle

tuned_agent = needle.Needle(
    tools=[weather_lookup, calendar_check],
    weights="tuned.cact"  # LoRA fine-tuned weights

)

response = tuned_agent.run("What's the weather in Tokyo?")
print(response.get("confidence"))

# → None

# Console warning on first instantiation:

# "finetuning does not update the confidence head, so scores are

#  uncalibrated for tuned weights; this agent reports confidence as None"

```

## Documentation References

The Needle documentation explicitly covers this limitation in two key locations:

- **[`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md)** (lines 85-88): Explains under "What finetuning does not change → The confidence head" that the confidence head is untouched during LoRA fine-tuning and that the package disables confidence for tuned weights

- **[`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md)** (lines 68-73): Details in the *Confidence* paragraph that confidence calculation has calibration limits and describes the effect of fine-tuning on the confidence field

## What This Means for Production Deployments

When deploying a fine-tuned Needle model, you should:

- **Remove any logic dependent on confidence thresholds** from your application
- **Implement alternative quality assurance** (human-in-the-loop, secondary verification, or output consistency checks)
- **Document this limitation** for downstream consumers of your API
- **Consider re-calibration** only if you have substantial validation data and can train a custom confidence head

## Summary

- **Confidence scores are base-model only**: The `ConfidenceHead` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) never updates during LoRA fine-tuning
- **Automatic disablement**: [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) detects tuned weights and sets `confidence: None` with a runtime warning
- **Documentation verified**: Both [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) and [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) explicitly document this behavior
- **No workaround available**: You must use base weights if confidence scores are required for your use case

## Frequently Asked Questions

### Can I manually enable confidence scores for fine-tuned weights?

No. The Needle package explicitly prevents this because uncalibrated confidence scores would be actively harmful—overconfident wrong predictions are worse than acknowledged uncertainty. There is no configuration flag to override this safety mechanism.

### Will future versions support confidence calibration for fine-tuned models?

The current architecture treats the confidence head as frozen infrastructure. Adding calibration support for fine-tuned weights would require either fine-tuning the confidence head itself (with appropriate validation data) or implementing domain adaptation techniques, neither of which is on the current roadmap according to the [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) documentation.

### How can I assess prediction quality without confidence scores?

Several alternatives exist: implement **consistency checks** by running the same query with minor variations, use **token-level surprisal** from the underlying language model, add a **secondary verification step** with a different model, or establish **human review thresholds** based on query complexity or domain criticality.

### Does this limitation apply to all types of fine-tuning?

Currently, yes. The documentation specifically mentions LoRA adapters, but the underlying mechanism—confidence head immutability—would affect any fine-tuning approach that does not explicitly retrain the confidence head component defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).