What Happens to the Confidence Score During LoRA Fine-Tuning in Needle

Needle resets the confidence score to None during LoRA fine-tuning because the confidence head is frozen while only adapter weights are trained, leaving the original confidence parameters uncalibrated for the adapted model.

The Needle library implements a dedicated confidence head that predicts per-token confidence scores for base model generations. However, when you apply LoRA (Low-Rank Adaptation) fine-tuning, this confidence mechanism is intentionally disabled to prevent misleading outputs.

How LoRA Fine-Tuning Affects Model Components

LoRA fine-tuning injects trainable low-rank matrices into specific layers of a language model while keeping the majority of original weights frozen. In Needle's implementation, this selective update strategy creates a critical mismatch with the confidence system.

The Confidence Head Remains Frozen

The confidence head is instantiated separately from the main transformer layers. In needle/model/architecture.py, the architecture defines:

self.confidence_head = ConfidenceHead(cfg.jax_dtype)

This head appears at line 488 and receives hidden states through forward_confidence (lines 572-576). Because LoRA training only modifies adapter matrices—not the base model structure—the ConfidenceHead parameters stay completely untouched during fine-tuning.

Why Confidence Scores Become None

After LoRA adaptation, the language model weights shift to a new distribution, but the confidence head retains its original (pre-tuning) parameters. This creates a calibration mismatch: the confidence head was trained to interpret hidden states from the base model, not the adapted model.

Needle handles this explicitly. As documented in needle/__init__.py (lines 60-61), the library warns users that "finetuning does not update the confidence head, so scores are uncalibrated for tuned weights; this agent reports confidence as None."

Verifying Confidence Behavior in Code

You can observe this behavior when running inference with a merged LoRA adapter:

import needle

# Load base model and merge LoRA weights

model = needle.load("meta-llama/Meta-Llama-3-8B-Instruct")
model = model.merge_lora_adapter("my_lora_adapter")

# Generate with adapted weights

response = model.generate("Explain quantum computing in simple terms.")

print(response["text"])           # Normal generation output

print(response.get("confidence")) # → None (explicitly disabled)

The confidence field returns None rather than a numeric score, protecting users from relying on uncalibrated confidence estimates.

Architecture Implications

Component	LoRA Training?	Post-Tuning State
Attention/FFN adapter matrices	Yes	Updated
Base transformer weights	No	Frozen original
ConfidenceHead parameters	No	Frozen original (stale)
Confidence output	N/A	None

This design prioritizes safety over convenience. Rather than exposing potentially misleading confidence values, Needle opts for transparent omission.

Recalibrating Confidence After LoRA

To restore confidence scores, you would need to:

1. Collect hidden states from the LoRA-adapted model on a representative dataset
2. Retrain or fine-tune the ConfidenceHead on these new hidden distributions
3. Merge the recalibrated head with your adapted model

No automatic recalibration is currently implemented in the Needle codebase.

Summary

• Confidence head freeze: The ConfidenceHead in needle/model/architecture.py receives no gradient updates during LoRA training
• Intentional None return: needle/__init__.py explicitly documents and enforces this behavior to prevent uncalibrated score usage
• Calibration gap: Original confidence parameters cannot interpret adapted model hidden states reliably
• Safety-first design: Needle opts for transparent omission rather than misleading confidence values

Frequently Asked Questions

Why can't Needle just keep using the original confidence scores after LoRA?

The confidence head was trained to interpret hidden state patterns from the base model. After LoRA adaptation, those hidden states shift to a new distribution. The frozen confidence head would systematically miscalibrate—typically overestimating or underestimating confidence in ways that are hard to predict. Needle returns None to signal this uncertainty explicitly.

Can I recalibrate the confidence head myself after LoRA fine-tuning?

Yes, but no built-in utilities exist in Needle currently. You would extract hidden states from your LoRA-adapted model on a validation set, then train the ConfidenceHead (defined at needle/model/architecture.py:488) on those adapted representations before reattaching it to your model.

Does this limitation apply to full fine-tuning or other adapter methods?

The analysis focuses on LoRA specifically. Full fine-tuning would update all parameters including the confidence head if included in the training loop. Other adapter methods (IA³, adapters) would require case-by-case verification of whether ConfidenceHead parameters are touched—check needle/model/architecture.py and your specific training configuration.

Where is the confidence head actually used during generation?

The forward_confidence method at needle/model/architecture.py:572-576 routes hidden cells to self.confidence_head. This path is conditionally skipped when LoRA weights are active, as enforced by the initialization logic in needle/__init__.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}