How Needle Selects the Top 5 Tools Per Turn: A Deep Dive into the Tool Selection Pipeline

Needle selects the top S tools per turn by parsing token-level probability scores from the LLM's tool call suggestions, sorting candidates in descending order, and keeping the first S entries (default 1, configurable via NEEDLE_TOP_S).

Needle is an open-source agent framework that orchestrates LLM-driven tool use. The core question of how Needle selects the top 5 tools per turn—or any number S—involves a precise pipeline that transforms Python functions into OpenAI-compatible schemas, prompts the model, and filters by probability. This article explains each step using the actual source code from the cactus-compute/needle repository.

How the @tool Decorator Prepares Functions for Selection

Every function that Needle can invoke must be decorated with @tool. This decorator, defined in needle/agent/tools.py (lines 62–64), attaches a JSON schema to the function that the LLM can understand.

def tool(fn):
    fn._needle_tool = build_schema(fn)      # ← creates the JSON schema

    return fn

The build_schema(fn) call introspects the function's signature, type hints, and docstring to generate an OpenAI-compatible tool definition. This schema includes the function name, parameter types, and descriptions.

Example: Registering a Search Tool

from needle import tool, Field

@tool
def search(query: str, num_results: int = Field(default=5, ge=1, le=10)):
    """Search the web for *query* and return up to *num_results* results."""
    # ... implementation omitted ...

# Verify the schema was attached

assert hasattr(search, "_needle_tool")
print(search._needle_tool["name"])            # → "search"

print(search._needle_tool["parameters"])      # JSON schema sent to LLM

Without this decorator, Needle cannot expose a function to the LLM, and it therefore cannot participate in the top S selection process.

The Turn Execution Flow in fetch.py

The actual selection logic resides in needle/agent/fetch.py. When run_turn() is called, the agent executes a three-phase pipeline:

1. Prompt construction — aggregates all tool schemas from registered functions
2. LLM inference — sends the prompt and receives candidate tool calls with log-probabilities
3. Scoring and truncation — sorts by probability and keeps the top S

Phase 1: Sending Tool Schemas to the LLM

The agent collects every function's _needle_tool attribute and includes these schemas in the chat completion request. The LLM generates a list of possible tool calls, where each call carries a token-level probability representing the model's confidence.

Phase 2: Parsing and Scoring Candidate Calls

After the LLM responds, Needle parses the raw output into structured candidates. Each candidate contains:

• The function name to invoke
• The parsed arguments
• A probability score extracted from the model's log-probabilities

Phase 3: Selecting the Top S Tools Per Turn

Here's where the core selection happens. Needle sorts all candidates by their probability scores in descending order and slices the list to retain only the first S entries:

S = int(os.getenv("NEEDLE_TOP_S", 1))   # default: 1, override via environment

top_candidates = sorted(candidates, key=lambda c: c.score, reverse=True)[:S]

The NEEDLE_TOP_S environment variable controls this threshold without code changes. To select the top 5 tools per turn instead of the default 1:

import os
os.environ["NEEDLE_TOP_S"] = "5"

from needle.agent.fetch import run_turn

result = run_turn(user_input="Analyze these three datasets and summarize findings")

# `result` contains up to 5 tool calls, ordered by LLM confidence

Configuration: Adjusting S Without Code Changes

The NEEDLE_TOP_S variable provides runtime flexibility. Common configurations include:

Value	Use Case
1 (default)	Single-tool deterministic workflows
3–5	Exploring multiple promising paths before committing
10+	Breadth-first search with heavy post-filtering

Set this before importing Needle components to ensure fetch.py reads the updated value:

export NEEDLE_TOP_S=5
python my_agent.py

Why Probability-Based Selection Matters

Needle's approach differs from simple random sampling or hard-coded priority lists. By respecting the LLM's own confidence estimates, the framework:

• Surfaces the most semantically relevant tools given the current conversation context
• Allows emergent behavior where the model "votes" for multiple complementary actions
• Provides transparency via inspectable probability scores

The scores are not post-hoc heuristics—they're the model's internal activation probabilities exposed through the API's logprobs feature.

Summary

• The @tool decorator in needle/agent/tools.py attaches JSON schemas to functions, making them selectable.
• needle/agent/fetch.py orchestrates turn execution: prompting, parsing, sorting by probability, and truncating to the top S.
• S defaults to 1 but is configurable via the NEEDLE_TOP_S environment variable.
• Selection relies on LLM-provided probability scores, not external ranking heuristics.
• To select the top 5 tools per turn, set NEEDLE_TOP_S=5 before calling run_turn().

Frequently Asked Questions

How does Needle handle ties when multiple tools have the same probability?

Needle uses Python's stable sort, so ties preserve the original order returned by the LLM. In practice, exact probability collisions are rare due to floating-point precision in log-probability calculations. No additional tie-breaking logic is implemented in the current source.

Can I change the top S limit per turn programmatically instead of via environment variable?

Currently, NEEDLE_TOP_S is read once from the environment when fetch.py loads. For per-call flexibility, you would need to modify run_turn() to accept an optional top_s parameter or manipulate os.environ immediately before invocation. The repository's CLI entry point in needle/cli.py does not expose this as a command-line flag.

What happens if the LLM suggests fewer than S tools?

Needle returns all available candidates without error. The slice operation [:S] gracefully handles shorter lists, so if the model proposes 2 tools and NEEDLE_TOP_S=5, both are returned. No padding or placeholder logic is applied.

Does Needle support weighted scoring beyond raw LLM probabilities?

As of the analyzed version, no. The selection pipeline uses the LLM's log-probabilities directly without learned re-ranking or human-configured weights. Custom scoring would require modifying the candidate sorting logic in needle/agent/fetch.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}