Needle 2 Confidence Score: Post‑Hoc Head and Decoding Probability Signals Explained

Needle 2 calculates confidence scores using two signals—a learned neural confidence head and the token‑level decoding probability—taking the minimum of the two to ensure conservative, reliable tool‑calling decisions.

Needle 2, the open‑source tool‑calling framework from Cactus Compute, provides a calibrated confidence score for every generated function call. Understanding how this score works is essential for building reliable agent systems that can gate low‑confidence actions. This article breaks down the two signals used for confidence scoring in Needle 2, explains how they combine, and shows practical implementation patterns from the source code.

The Two Confidence Signals in Needle 2

Needle 2's confidence mechanism relies on two independent signals. The final reported value is the minimum of these two, ensuring that a call passes only when both sources agree on high confidence.

Signal 1: Post‑Hoc Confidence Head

The post‑hoc confidence head is a learned neural module named ConfidenceHead that evaluates the complete prompt together with the generated function call. It produces a calibrated probability reflecting how well the model's output matches the training distribution.

In needle/model/architecture.py, the confidence head is implemented as follows:

# Lines 63-75 in needle/model/architecture.py

class ConfidenceHead(nn.Module):
    def __init__(self, hidden_size: int):
        super().__init__()
        self.dense = nn.Linear(hidden_size, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # Pool final hidden state and predict confidence

        pooled = hidden_states[:, -1, :]
        return self.sigmoid(self.dense(pooled)).squeeze(-1)

This head is trained only on the base model weights. When you load a fine‑tuned agent with LoRA adapters, the confidence head becomes unavailable and returns None—a warning is emitted at construction time in needle/__init__.py (lines 60-63).

Signal 2: Decoding Probability of Call Tokens

The decoding probability signal captures the token‑level likelihood that the model assigned while generating the JSON‑structured function call. This is derived directly from the model's softmax outputs during autoregressive decoding, representing the model's own certainty about each generated token.

This signal requires no additional training and remains available even with fine‑tuned adapters.

How Needle 2 Combines the Two Signals

According to the API documentation in doc/apis.md (lines 51-54), Needle 2 applies a conservative policy: the final confidence is the lower of the two signals. This design guarantees that:

• A call is accepted only when both the learned head and token‑level probability indicate reliability
• Downstream systems can implement hard gates (e.g., refusing calls below a threshold)
• False positives are minimized in production tool‑calling pipelines

Handling Fine‑Tuned Agents and Missing Confidence

Because the ConfidenceHead is tied to base model weights, fine‑tuned agents present a special case. The runtime logic in needle/__init__.py handles this:

# Lines 60-63 in needle/__init__.py

if self.using_lora:
    warnings.warn(
        "Confidence scores unavailable with fine-tuned weights; "
        "head was trained on base model only."
    )
    self.confidence_head = None

When confidence_head is None, the confidence field in responses will be None. Your application code should handle this gracefully.

Practical Implementation: Gating Calls by Confidence

The following pattern demonstrates how to integrate confidence scoring into your Needle 2 application:

import needle

# Declare a simple tool

@needle.tool
def set_lights(room: str, on: bool, brightness: int = 0):
    """Control lights in a room."""
    return {"room": room, "on": on, "brightness": brightness}

# Create an agent with a confidence gate of 0.7

agent = needle.Needle(tools=[set_lights])

# Run a query

response = agent.run("Turn on the kitchen lights to 50 percent")

# Inspect the confidence score

conf = response.get("confidence")
print("Confidence:", conf)

# Optionally reject low‑confidence calls

if conf is not None and conf < 0.7:
    print("⚠️ Low confidence – you may want to re‑ask or fallback to a larger model.")
else:
    print("✅ High confidence – safe to act on the result.")

Running Tests with Confidence Thresholds

The built‑in test harness in needle/environments/_harness.py (lines 28-33) supports filtering by minimum confidence:

from needle.environments import smart_home

# Run the built‑in test suite; set a minimum confidence of 0.5

smart_home.run_tests(min_confidence=0.5)

This allows systematic evaluation of model reliability across your tool set.

Summary

• Two signals drive Needle 2 confidence scoring: the learned ConfidenceHead post‑hoc evaluator and the raw token‑level decoding probability.
• Conservative combination: the final score is the minimum of the two signals, accepting calls only when both sources agree.
• Base model dependency: the confidence head requires base model weights and returns None for LoRA‑fine‑tuned agents.
• Source locations: ConfidenceHead implementation in needle/model/architecture.py (lines 63-75), combination logic in doc/apis.md (lines 51-54), runtime handling in needle/__init__.py (lines 60-63), and test filtering in needle/environments/_harness.py (lines 28-33).

Frequently Asked Questions

What are the exact two signals used for confidence scoring in Needle 2?

Needle 2 uses a post‑hoc confidence head (a learned neural network that evaluates prompt‑plus‑call combinations) and the decoding probability of the generated call tokens (the token‑level softmax probabilities during generation). The final confidence is the minimum of these two values.

Why does Needle 2 take the minimum of the two confidence signals?

Taking the minimum implements a conservative reliability policy. A call receives high confidence only when both the trained head and the model's own generation probability agree. This reduces false positives in critical tool‑calling scenarios where incorrect actions could have significant consequences.

Why is confidence unavailable with fine‑tuned Needle 2 models?

The ConfidenceHead is trained only on base model weights. When you load a fine‑tuned agent with LoRA adapters, the head's predictions no longer align with the modified distribution. Needle 2 explicitly disables the head in this case and emits a warning, returning None for confidence scores to avoid misleading calibration.

How can I filter Needle 2 test results by confidence level?

Use the min_confidence parameter in the environment test harness as shown in needle/environments/_harness.py. Pass a float threshold (e.g., 0.5 or 0.7) to smart_home.run_tests() or equivalent environment runners to exclude low‑confidence predictions from your evaluation results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}