When Does Tool Retrieval Engage Automatically in Needle: How It Works

Needle’s tool retrieval engages automatically at the start of every model turn when an agent is instantiated with tools, selecting the top five most relevant tools and constraining the decoder grammar to generate valid calls only from that subset.

The cactus-compute/needle repository implements an agentic framework where tool retrieval operates without explicit user intervention. When you invoke agent.run(), the system automatically evaluates the conversation context and query to determine which available tools are relevant for that specific turn. This automatic selection prevents the model from attempting to invoke irrelevant or out-of-scope functions while ensuring generated JSON remains well-formed.

Automatic Trigger Points for Tool Retrieval

Tool retrieval activates at the beginning of each model turn immediately upon calling agent.run(...).

You do not need to specify which tools to consider for each query. Instead, the framework inspects the user prompt and conversation history to determine relevance. This design ensures that the retrieval head—implemented in the model architecture—scores every tool in the declared catalog against the current context before any generation begins.

The Five-Step Retrieval Workflow

The automatic retrieval process follows a strict pipeline that constrains the model's output space to valid, relevant tool calls.

1. Catalog Declaration

First, you supply a list of tool functions when instantiating the agent. These are typically decorated with @needle.tool and include type-annotated signatures and docstrings that describe their purpose.

2. Retrieval Head Scoring

For each turn, the model computes relevance scores for every declared tool based on the current prompt and past interactions. According to the source code, this scoring mechanism lives in needle/model/architecture.py, which defines the retrieval head that evaluates semantic similarity between the query and available tool descriptions.

3. Top-K Selection

The system selects the top five highest-scoring tools and discards the rest. This top-K filtering is crucial for performance and accuracy, as it prevents the model from considering irrelevant options. As implemented in needle/agent/tools.py, this selection process masks all non-selected tools from the generation grammar.

4. Grammar Constrained Generation

Once the top five tools are selected, Needle compiles a byte-level grammar from their JSON schemas. This grammar constrains the decoder, forcing it to emit only valid JSON calls for the selected tools. The constraint logic ensures that even if the model attempts to hallucinate a tool name or parameter, the generated output remains structurally valid and limited to the retrieved subset.

5. Execution and Feedback Loop

After generation, the chosen tool calls are executed, and their results are fed back into the conversation context. The model then continues its response, potentially triggering another retrieval cycle if additional tool calls are required to complete the task.

Implementation Details

The core retrieval logic resides in needle/agent/tools.py, which handles the ranking, top-K selection, and grammar constraint compilation. The retrieval head itself is defined in needle/model/architecture.py, where it computes the relevance scores used to filter the tool catalog.

This architecture ensures that tool retrieval is zero-config—simply providing a tools list to needle.Needle() enables the automatic filtering mechanism. The README documentation confirms that this behavior occurs "without any explicit user request" beyond the initial agent instantiation.

Code Example: Automatic Retrieval in Action

The following example demonstrates how retrieval runs automatically when processing a multi-tool request:

import needle

# 1️⃣ Declare a few tools

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

@needle.tool
def set_lights(room: str, brightness: int):
    """Set the brightness of lights in a room."""
    return {"room": room, "brightness": brightness}

# 2️⃣ Create an agent with the catalog

agent = needle.Needle(tools=[get_weather, set_lights])

# 3️⃣ Run a query – tool retrieval runs automatically

response = agent.run("Dim the living room lights to 20 percent and tell me if it's sunny in Paris")
print(response["results"])

# Example output:

# [{'room': 'living room', 'brightness': 20},

#  {'city': 'Paris', 'temp_c': 27, 'sky': 'clear'}]

In this execution, the model automatically selects set_lights and get_weather from the catalog (as the top-five relevant tools) before generating any calls. The grammar constraint ensures the output conforms to the schemas of only these selected tools.

Summary

• Tool retrieval triggers automatically at the beginning of every agent.run() call, requiring no manual configuration per query.
• The system evaluates all declared tools against the current conversation context using a retrieval head defined in needle/model/architecture.py.
• Only the top five tools are retained for generation, with all others masked from the decoder.
• Grammar constraints compiled from the selected tool schemas ensure valid JSON output exclusively for the retrieved subset.
• The implementation is contained primarily in needle/agent/tools.py, with high-level documentation in the repository's README.md.

Frequently Asked Questions

When exactly does tool retrieval run during inference?

Tool retrieval runs immediately when you invoke agent.run(), specifically at the start of each model turn before any token generation begins. The system evaluates the user query and conversation history to score tools before the decoder produces output.

How many tools does Needle select automatically?

Needle automatically selects the top five tools based on relevance scores computed by the retrieval head. This top-K selection is hardcoded into the retrieval mechanism to balance context window efficiency with functional coverage.

Can I disable automatic tool retrieval or adjust the number of selected tools?

Based on the current implementation in needle/agent/tools.py, automatic retrieval is the default behavior when tools are provided at agent instantiation. To bypass retrieval, you would need to instantiate the agent without a tools catalog. The selection limit of five tools appears to be a fixed constraint in the current architecture.

Where is the retrieval head implemented in the codebase?

The retrieval head is implemented in needle/model/architecture.py, which defines the model blocks responsible for scoring tool relevance. The selection logic, top-K filtering, and grammar constraint compilation are handled in needle/agent/tools.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}