Needle 2 Tool Retrieval: Scaling to Thousands of Tools with Contrastive Embeddings

Needle 2 indexes every declared tool once using a built-in contrastive embedding head, then retrieves only the top 5 most relevant tools per user query to keep inference efficient with large catalogs.

The cactus-compute/needle repository implements a sophisticated retrieval system that allows AI agents to work with hundreds or thousands of tools without overwhelming the model's context window. Instead of injecting every available function into each prompt, Needle 2 uses needle 2 tool retrieval to select relevant capabilities before inference begins.

The Challenge of Large Tool Catalogs

Large language models have finite context windows. When an agent declares hundreds of tools—each with its own JSON schema—attempting to include them all in every prompt would exhaust token limits and degrade performance. Needle 2 solves this by treating tool selection as a retrieval problem rather than a context-window management problem.

How Needle 2 Tool Retrieval Works

The retrieval pipeline operates in three distinct phases: initial indexing, persistent storage, and per-query selection.

One-Time Schema Embedding

When a Needle instance is created, the engine processes every tool immediately. In needle/__init__.py (lines 54-66), the __init__ method passes each tool's JSON schema to the native engine (needle_init). The engine's built-in contrastive head embeds every tool schema a single time, converting structural definitions into dense vector representations suitable for similarity comparison.

This indexing happens once during initialization, not during inference, ensuring that the computational cost is paid upfront rather than on every request.

Query-Driven Top-5 Selection

For every user turn, Needle 2 embeds the query text and computes similarity scores against the pre-computed tool embeddings. According to the documentation in doc/apis.md, the system selects only the top-5 scoring tools for inclusion in that turn's prompt grammar.

This is a hard constraint: unselected tools are unreachable for that specific step, not merely unlikely to be chosen. The retrieval step occurs outside the language model, so the LLM never sees the full catalog. This design keeps token consumption predictable and latency consistent regardless of catalog size.

Persistent Embedding Cache

Computing embeddings for massive catalogs can be expensive. Needle 2 offers the tool_index_path parameter in Needle.__init__ to persist embeddings on disk. The engine stores vectors keyed by a fingerprint of the tool schemas combined with the model version.

When loading:

• Matching fingerprints load instantly from disk
• Changed schemas trigger re-embedding of only the modified entries

This incremental update mechanism makes subsequent initializations nearly instantaneous for unchanged catalogs.

Implementing Tool Retrieval in Practice

The following example demonstrates declaring a large catalog, persisting embeddings, and querying the agent:

from needle import Needle, tool

# Declare a large catalog (more than five tools)

big_catalog = [
    # Simple functions decorated with @tool

    *[lambda x, i=i: f"Result {i}" for i in range(20)],   # 20 dummy tools

]

# Persist embeddings so the first run does the heavy work only once

agent = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",   # ← disk cache for embeddings

)

# Ask a question that only a few tools are relevant to

response = agent.run(
    "Which tool returns the result for i=7?",
    max_steps=1,
)

print(response["function_calls"])

# → will contain a call only to the tool whose embedding best matches the query

To reuse the cached index in a new process, simply provide the same path:

# Re-using the same index in a new process (instant load)

agent2 = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",   # loads the pre-computed embeddings

)

print(agent2.run("Give me the result for i=13")["function_calls"])

The schema definitions are handled by needle/agent/tools.py, which provides the @tool decorator and conversion functions (build_schema, pydantic_schema) that transform Python callables into JSON-schema tool definitions compatible with the embedding engine.

Summary

• Contrastive embedding: Needle 2 embeds tool schemas once at initialization using a dedicated head in the native engine.
• Hard top-5 limit: Only the five highest-similarity tools enter the context per turn; the rest are unreachable.
• Persistent storage: The tool_index_path parameter enables disk-based caching with fingerprint-based invalidation.
• External retrieval: Tool selection happens outside the LLM, preserving context window for actual reasoning.

Frequently Asked Questions

Why does Needle 2 limit retrieval to five tools?

The number five represents a design trade-off that balances context window constraints against model capability. According to the source documentation, five tools fit comfortably within the model's context while still providing sufficient optionality for complex queries. This limit ensures predictable token usage and consistent latency regardless of total catalog size.

How does the tool_index_path cache work?

The cache stores embeddings on disk keyed by a cryptographic fingerprint of the tool schemas combined with the model version identifier. When initializing with tool_index_path, the engine compares the current fingerprint against the stored version. A matching fingerprint loads instantly, while schema modifications trigger selective re-embedding of only the changed entries.

Is the top-5 limit configurable in the source code?

The documentation and source analysis indicate that five is a fixed retrieval limit hardcoded in the retrieval logic. The constraint is described as a hard boundary where unselected tools become unreachable rather than improbable, suggesting this is a core architectural decision rather than a user-configurable parameter in the current implementation.

Where does tool schema generation occur?

Tool schemas are generated in needle/agent/tools.py, which implements the @tool decorator and utility functions like build_schema and pydantic_schema. These components convert Python callables into JSON-schema definitions that the contrastive head in needle/__init__.py subsequently embeds during initialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}