How Confidence Scores and Threshold Gating Work in Needle 2: Implementation Guide

Needle 2 computes per-generation confidence scores by pooling transformer hidden states through a learned ConfidenceHead module, exposing the resulting logit to users for custom threshold-based gating.

Needle 2, an open-source inference engine developed by cactus-compute, provides built-in confidence estimation for language model outputs. This article examines how the ConfidenceHead module generates scores, how the engine delivers them to Python callers, and why threshold gating remains entirely user-controlled.

Confidence Score Architecture

The ConfidenceHead Module

At the core of Needle 2's confidence system is the ConfidenceHead class, defined in needle/model/architecture.py (lines 63-76). This lightweight nn.Module performs two operations:

1. Pools hidden states using a learned probe matrix
2. Projects the pooled vector to a single scalar logit

The confidence head attaches to the main transformer and receives hidden representations after the model processes input tokens.

forward_confidence Method

The forward_confidence method (lines 72-77 in needle/model/architecture.py) implements the public-facing confidence computation:

# Simplified conceptual flow based on source structure

def forward_confidence(self, hidden_cells, attention_mask):
    # Mask out padding tokens

    masked_hidden = hidden_cells * attention_mask.unsqueeze(-1)
    # Feed to ConfidenceHead for pooling + projection

    confidence_logit = self.confidence_head(masked_hidden)
    return confidence_logit

This method is invoked by the engine whenever confidence measurement is requested.

Engine Integration and API Response

C-Extension Delivery

The Needle engine's C-extension receives the confidence logit and packages it into a JSON envelope returned to the Python wrapper. The Needle.complete() method in needle/__init__.py (lines 124-125) surfaces this value:

from needle import Needle

agent = Needle()

# Completion includes confidence score

response = agent.complete("What is the capital of Finland?")
print(response["text"])        # "Helsinki"

print(response["confidence"])  # 0.94 (example logit value)

Finetuned Weights Warning

A critical edge case occurs when users load custom finetuned weights. As implemented in needle/__init__.py (lines 59-62), the confidence head is not retrained during fine-tuning. The wrapper detects this condition and deliberately nulls the score:

# Loading finetuned weights triggers warning and null confidence

agent = Needle(weights="custom_finetuned.cact")

# Warning emitted: "Confidence scores uncalibrated for finetuned weights"

response = agent.complete("Any prompt")
print(response["confidence"])  # None

This prevents misleading confidence values when the head's calibration no longer matches the base model.

Threshold Gating: User-Side Implementation

Needle 2 does not enforce hard thresholds internally. The library returns raw confidence logits, leaving gating policy decisions to downstream applications. This design maintains flexibility across diverse use cases.

Custom Threshold Example

from needle import Needle

agent = Needle()
CONFIDENCE_THRESHOLD = 0.8

response = agent.complete("Explain Riemannian geometry")

# Implement application-specific gating

if response["confidence"] is None:
    print("Warning: No confidence available (finetuned weights?)")
elif response["confidence"] < CONFIDENCE_THRESHOLD:
    print(f"Low confidence ({response['confidence']:.2f}) — escalating to human review")
else:
    print(f"High confidence ({response['confidence']:.2f}) — accepting automatically")

Common gating patterns include:

• Binary acceptance: Accept above threshold, reject below
• Tiered routing: High → auto-accept, medium → human review, low → reject
• Dynamic thresholds: Adjust based on query complexity or user history

Supporting Infrastructure

Model Export Registration

The confidence head is registered in needle/model/export.py via HEAD_CODES, ensuring proper serialization when exporting trained models.

Test Coverage

The tests/test_weights.py file contains sanity checks verifying that API responses include the confidence field with expected value ranges.

Summary

• ConfidenceHead in needle/model/architecture.py pools hidden states and projects to a scalar logit
• forward_confidence masks padding and invokes the head for per-generation scoring
• Finetuned weights trigger automatic confidence nulling with user warning (lines 59-62, 123-125 in needle/__init__.py)
• Threshold gating is entirely user-implemented; Needle returns raw logits for maximum flexibility

Frequently Asked Questions

What happens to confidence scores when I use custom finetuned weights?

The confidence head remains frozen when loading finetuned weights. As implemented in needle/__init__.py, the wrapper detects non-default weights and sets confidence to None while emitting a calibration warning. This prevents unreliable scores from mismatched head weights.

Can I retrain the ConfidenceHead for my finetuned model?

The current Needle 2 implementation does not support confidence head retraining through the public API. Users seeking calibrated scores for custom weights would need to modify needle/model/architecture.py and add head training to their fine-tuning pipeline.

What range do confidence logits use?

Needle 2 returns unbounded scalar logits, not normalized probabilities. Typical values fall between -10 and 10, but thresholds should be calibrated against your specific dataset rather than assuming fixed ranges.

Does threshold gating affect generation quality?

No—threshold gating occurs after generation completes. The confidence score reflects the model's internal uncertainty about its already-produced output, not a decision made during token sampling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}