Needle Tool Retrieval Head: How Top‑S (Top 5) Tool Selection Works

Needle’s tool‑retrieval head filters large tool catalogs down to the top‑S (default 5) most relevant candidates per turn, constraining the decoder grammar so that generated JSON calls are guaranteed to match one of the selected schemas.

The cactus‑compute/needle repository implements a compact 45 M‑parameter foundation model specifically engineered for efficient tool‑calling on resource‑constrained devices. At the heart of its efficiency lies the Needle tool retrieval head top 5 tools selection mechanism, which dynamically narrows potentially massive tool catalogs to a manageable subset before generation begins. This approach reduces memory pressure and ensures syntactically valid outputs without requiring external API calls or massive context windows.

How the Tool‑Retrieval Head Works in Needle

The retrieval head operates as an internal attention‑based scoring mechanism that evaluates every declared tool against the current user query. Unlike retrieval‑augmented generation systems that fetch external documents, Needle’s retrieval head operates entirely within the 28 MB RAM footprint of the model.

Scoring Tools with Attention Mechanisms

In needle/model/architecture.py, the retrieval head computes relevance scores by applying GQA‑style attention over tool descriptions, names, and argument schemas. Each tool in the catalog receives a continuous score representing its probability of being relevant to the current turn. The model processes these embeddings through Sinkhorn‑normalised routing, ensuring stable gradient flow during inference.

Enforcing the Top‑S Constraint

Once scores are computed, the head emits a softmax distribution across the tool catalog. The top‑S (where S defaults to 5) highest‑scoring tools are retained as a kv‑sink—a fixed memory buffer that the decoder grammar references during generation. As implemented in needle/agent/tools.py, this selection happens before token generation begins, effectively pruning the search space so that the model only "sees" relevant tools.

Grammar‑Constrained Decoding and Syntactic Guarantees

Because the selected top‑S tools are baked into the decoder grammar, Needle guarantees that any generated JSON strictly conforms to one of the retained schemas. This constraint eliminates hallucinated tool names or malformed arguments. The decoder uses the kv‑sink to validate tokens against the selected tool signatures in real time, ensuring that even on a 256‑token sliding window, the output remains structurally valid.

Confidence‑Gated Output for Safe Execution

Every tool call includes a calibrated confidence score produced by a dedicated confidence head. This scalar value ∈ [0, 1] predicts the model’s certainty about the retrieved tools and generated parameters. Users can configure automatic execution thresholds; responses falling below the threshold trigger manual review pipelines, adding a safety layer to the top‑S selection process.

Architecture Underpinnings

The retrieval head relies on Needle’s Simple Attention Network architecture, which replaces traditional feed‑forward layers with Hadamard MLPs for fast n log n matrix‑free computation. Combined with GQA‑style attention and engram KV memory, this design allows the retrieval head to fetch tool embeddings efficiently without exceeding the 14 MB engine size (.cact file). The entire system runs as a single cached binary fetched from Hugging Face, requiring no external model files.

Implementing Top‑S Tool Selection in Practice

To use the retrieval head in your application, declare tools using the @needle.tool decorator and pass them to the Needle agent. The model automatically handles top‑S filtering during the run() invocation.

import needle

# Declare tools—the model will internally score these and keep only the top‑S

@needle.tool
def set_lights(room: str, brightness: int):
    """Set the lights in a given room to a specific brightness (0‑100)."""
    return {"room": room, "brightness": brightness}

@needle.tool
def dim_music(volume: int):
    """Lower the music volume."""
    return {"volume": volume}

# Build the agent with the full tool list

agent = needle.Needle(tools=[set_lights, dim_music])

# Execute—the retrieval head selects the most relevant tools (top‑S=5 default)

# and the decoder generates constrained JSON matching only those schemas

response = agent.run("Dim the kitchen lights to 20 and lower the music.")
print(response["results"])

# → [{'room': 'kitchen', 'brightness': 20}, {'volume': 30}]

In this example, needle/agent/tools.py processes the schema definitions, while needle/model/run.py orchestrates the inference loop that applies the confidence gating and retrieval scoring.

Key Source Files for Tool Retrieval

Understanding the implementation requires examining these specific files:

• needle/agent/tools.py – Implements the tool‑retrieval head logic and top‑S selection constraints
• needle/model/architecture.py – Defines the Simple Attention Network and retrieval head attention mechanisms
• needle/model/run.py – Contains the core inference loop handling confidence scoring and grammar enforcement
• needle/__init__.py – Exposes the public API including the @needle.tool decorator and Needle class

Summary

• The tool‑retrieval head scores all available tools using internal GQA‑style attention over tool descriptions and schemas.
• By default, only the top‑S (5) candidates are retained as a kv‑sink, drastically reducing memory usage and search space.
• The decoder grammar is constrained to these selected tools, guaranteeing syntactically valid JSON outputs.
• A confidence head provides calibrated certainty scores for each tool call, enabling threshold‑based automated execution.
• All functionality runs within a 14 MB engine using Hadamard MLPs and requires no external dependencies beyond the cached .cact file.

Frequently Asked Questions

What does “top‑S” mean in Needle’s tool retrieval system?

Top‑S refers to the subset of highest‑scoring tools selected by the retrieval head for a given turn. By default, S equals 5, meaning the model considers only the five most relevant tools from the full catalog when generating JSON calls, ensuring efficient computation on small devices.

How does Needle guarantee valid JSON output after selecting top‑S tools?

After selecting the top‑S tools, Needle bakes these schemas into the decoder grammar as a kv‑sink. This constraint forces the tokenizer to emit tokens that conform exclusively to one of the selected tool signatures, eliminating syntax errors or hallucinated tool names.

Where is the tool‑retrieval head implemented in the Needle codebase?

The retrieval head logic resides primarily in needle/agent/tools.py for the selection and schema preparation, while needle/model/architecture.py defines the attention mechanisms that perform the actual scoring of tool relevance against user queries.

Can I adjust the number of tools selected by the retrieval head?

While the default top‑S value is 5, the system is designed to allow configuration of this parameter during agent initialization. Adjusting S trades off between recall (more tools considered) and computational efficiency (larger kv‑sink and search space).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}