# Why Confidence Reports None with Fine-Tuned Weights in Needle and How to Handle It

> Understand why Needle's confidence reports None with fine-tuned weights and learn how to fix it. Discover solutions for misleading scores when the calibration head isn't updated.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-14

---

**`confidence` returns `None` for fine-tuned Needle models because the calibration head is not updated during fine-tuning, making any score potentially misleading.**

When you load a fine-tuned checkpoint in Needle—such as one with LoRA adapters—the **confidence head remains frozen** while only the language model weights change. This design protects you from over- or under-confident predictions, but it requires understanding how to work around the missing scores when you need them.

## Why Fine-Tuned Weights Disable Confidence Scoring

The `ConfidenceHead` in Needle is trained separately on the base model's output distribution. Fine-tuning with techniques like LoRA modifies that distribution without touching the head, breaking the calibration. Returning a number would be worse than returning nothing.

In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the constructor detects user-supplied weights and explicitly forces `response["confidence"] = None`:

```python

# From needle/__init__.py#L58-L60

if self.has_custom_weights:
    import warnings
    warnings.warn("Confidence scores are not available with fine-tuned weights; "
                  "the confidence head remains uncalibrated.")
    self._confidence_available = False

```

The same limitation is documented in `needle/model/finetune.py#L99-L102` where the CLI notes: *"confidence reports None with tuned weights; the head is not tuned."*

## How to Handle Missing Confidence Scores

Choose your approach based on whether you need probability estimates:

### Option 1: Ignore Confidence If Not Required

For deterministic tasks—classification, extraction, or tool calling where you only need the answer—simply omit confidence from your logic:

```python
from needle import Needle

tuned = Needle(weights="model_lora.cact")
response = tuned.complete("Extract the date from this invoice.")
print(response["text"])  # Valid output

# response["confidence"] is None — safe to ignore

```

### Option 2: Use Base Weights for Calibrated Confidence

Run inference twice: once with fine-tuned weights for quality, once with base weights for the score:

```python
base = Needle(weights="model.cact")
tuned = Needle(weights="model_lora.cact")

def complete_with_confidence(prompt):
    # Get high-quality response from tuned model

    tuned_resp = tuned.complete(prompt)
    
    # If confidence missing, re-query base model

    if tuned_resp.get("confidence") is None:
        base_resp = base.complete(prompt)
        tuned_resp["confidence"] = base_resp.get("confidence")
    
    return tuned_resp

result = complete_with_confidence("Summarize this legal document.")
print(f"Answer: {result['text'][:100]}...")
print(f"Confidence: {result['confidence']:.2f}")

```

### Option 3: Implement a Fallback Heuristic

Substitute `None` with a custom estimate based on available signals:

```python
def estimate_confidence(response, model=None):
    """Fallback confidence estimation when calibration is unavailable."""
    if response.get("confidence") is not None:
        return response["confidence"]
    
    # Heuristic: use response length as proxy (shorter = more confident)

    text = response.get("text", "")
    token_count = len(text.split())
    return max(0.0, 1.0 - (token_count / 100))  # Simple inverse scaling

# Usage

response = tuned.complete("Translate to French: Hello world.")
print(f"Estimated confidence: {estimate_confidence(response):.2f}")

```

### Option 4: Suppress Warnings in Test Suites

Capture and silence the calibration warning when testing:

```python
import warnings
from needle import Needle

def test_finetuned_model():
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        model = Needle(weights="model_lora.cact")
        # ... test assertions ...

```

## Architecture Reference: Where the Logic Lives

| File | Purpose |
|------|---------|
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Constructor warning and `None` enforcement for custom weights |
| [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) | CLI documentation of the limitation post-LoRA build |
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | `ConfidenceHead` definition (lines 488–576); unchanged during fine-tuning |

The `ConfidenceHead` remains at its pre-training state because fine-tuning calls in [`finetune.py`](https://github.com/cactus-compute/needle/blob/main/finetune.py) only update adapter layers, never touching the separate calibration network.

## Re-Calibrating the Confidence Head (Advanced)

Currently, **re-training the confidence head is not exposed in the public API**. The head would need training data from the fine-tuned distribution to produce valid scores. Until this becomes available, the recommended paths are: use base weights, apply heuristics, or accept the absence of calibrated uncertainty.

## Summary

- **Root cause**: The `ConfidenceHead` is not updated during fine-tuning, so `confidence` returns `None` to prevent misleading scores.
- **Detection**: Check [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) constructor logic that warns and sets `_confidence_available = False`.
- **Workarounds**: Ignore the field, query base weights separately, implement heuristics, or suppress warnings.
- **No API yet**: Re-calibration requires internal training not currently exposed.

## Frequently Asked Questions

### What triggers the None confidence value in Needle?

Loading any checkpoint with modified weights—LoRA adapters, continued pre-training, or full fine-tuning—disables confidence. The constructor in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) checks `has_custom_weights` and forces the field to `None` regardless of fine-tuning method.

### Can I enable confidence scoring for my fine-tuned model?

Not through the public API. The `ConfidenceHead` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) requires re-training on outputs from your specific fine-tuned checkpoint. Until Cactus Compute exposes this functionality, you must use base weights or custom heuristics.

### Does the None confidence affect other response fields?

No. `response["text"]`, tool calls, and all other outputs remain fully valid. Only the calibrated probability estimate is withheld. Your application logic should treat `None` as "uncertainty unknown" rather than failure.

### How can I silence the calibration warning in production?

Wrap instantiation in `warnings.catch_warnings()` as shown in Option 4 above. In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the warning is emitted via the standard library, so standard Python warning filters apply.