# How Confidence Scores and Threshold Gating Work in Needle 2: Implementation Guide

> Learn how Needle 2 implements confidence scores and threshold gating. Discover how to use the ConfidenceHead module and custom thresholds for precise control in your transformer models.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-25

---

**Needle 2 computes per-generation confidence scores by pooling transformer hidden states through a learned `ConfidenceHead` module, exposing the resulting logit to users for custom threshold-based gating.**

Needle 2, an open-source inference engine developed by cactus-compute, provides built-in confidence estimation for language model outputs. This article examines how the `ConfidenceHead` module generates scores, how the engine delivers them to Python callers, and why threshold gating remains entirely user-controlled.

## Confidence Score Architecture

### The ConfidenceHead Module

At the core of Needle 2's confidence system is the **`ConfidenceHead`** class, defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 63-76). This lightweight `nn.Module` performs two operations:

1. **Pools hidden states** using a learned probe matrix
2. **Projects** the pooled vector to a single scalar logit

The confidence head attaches to the main transformer and receives hidden representations after the model processes input tokens.

### forward_confidence Method

The **`forward_confidence`** method (lines 72-77 in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)) implements the public-facing confidence computation:

```python

# Simplified conceptual flow based on source structure

def forward_confidence(self, hidden_cells, attention_mask):
    # Mask out padding tokens

    masked_hidden = hidden_cells * attention_mask.unsqueeze(-1)
    # Feed to ConfidenceHead for pooling + projection

    confidence_logit = self.confidence_head(masked_hidden)
    return confidence_logit

```

This method is invoked by the engine whenever confidence measurement is requested.

## Engine Integration and API Response

### C-Extension Delivery

The Needle engine's C-extension receives the confidence logit and packages it into a JSON envelope returned to the Python wrapper. The **`Needle.complete()`** method in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 124-125) surfaces this value:

```python
from needle import Needle

agent = Needle()

# Completion includes confidence score

response = agent.complete("What is the capital of Finland?")
print(response["text"])        # "Helsinki"

print(response["confidence"])  # 0.94 (example logit value)

```

### Finetuned Weights Warning

A critical edge case occurs when users load custom finetuned weights. As implemented in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 59-62), the confidence head **is not retrained** during fine-tuning. The wrapper detects this condition and deliberately nulls the score:

```python

# Loading finetuned weights triggers warning and null confidence

agent = Needle(weights="custom_finetuned.cact")

# Warning emitted: "Confidence scores uncalibrated for finetuned weights"

response = agent.complete("Any prompt")
print(response["confidence"])  # None

```

This prevents misleading confidence values when the head's calibration no longer matches the base model.

## Threshold Gating: User-Side Implementation

Needle 2 **does not enforce hard thresholds internally**. The library returns raw confidence logits, leaving gating policy decisions to downstream applications. This design maintains flexibility across diverse use cases.

### Custom Threshold Example

```python
from needle import Needle

agent = Needle()
CONFIDENCE_THRESHOLD = 0.8

response = agent.complete("Explain Riemannian geometry")

# Implement application-specific gating

if response["confidence"] is None:
    print("Warning: No confidence available (finetuned weights?)")
elif response["confidence"] < CONFIDENCE_THRESHOLD:
    print(f"Low confidence ({response['confidence']:.2f}) — escalating to human review")
else:
    print(f"High confidence ({response['confidence']:.2f}) — accepting automatically")

```

Common gating patterns include:

- **Binary acceptance**: Accept above threshold, reject below
- **Tiered routing**: High → auto-accept, medium → human review, low → reject
- **Dynamic thresholds**: Adjust based on query complexity or user history

## Supporting Infrastructure

### Model Export Registration

The confidence head is registered in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) via **`HEAD_CODES`**, ensuring proper serialization when exporting trained models.

### Test Coverage

The [`tests/test_weights.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_weights.py) file contains sanity checks verifying that API responses include the confidence field with expected value ranges.

## Summary

- **ConfidenceHead** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) pools hidden states and projects to a scalar logit
- **`forward_confidence`** masks padding and invokes the head for per-generation scoring
- **Finetuned weights** trigger automatic confidence nulling with user warning (lines 59-62, 123-125 in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py))
- **Threshold gating** is entirely user-implemented; Needle returns raw logits for maximum flexibility

## Frequently Asked Questions

### What happens to confidence scores when I use custom finetuned weights?

The confidence head remains frozen when loading finetuned weights. As implemented in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the wrapper detects non-default weights and sets `confidence` to `None` while emitting a calibration warning. This prevents unreliable scores from mismatched head weights.

### Can I retrain the ConfidenceHead for my finetuned model?

The current Needle 2 implementation does not support confidence head retraining through the public API. Users seeking calibrated scores for custom weights would need to modify [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and add head training to their fine-tuning pipeline.

### What range do confidence logits use?

Needle 2 returns unbounded scalar logits, not normalized probabilities. Typical values fall between -10 and 10, but thresholds should be calibrated against your specific dataset rather than assuming fixed ranges.

### Does threshold gating affect generation quality?

No—threshold gating occurs **after** generation completes. The confidence score reflects the model's internal uncertainty about its already-produced output, not a decision made during token sampling.