How Needle 2 Handles Large Tool Catalogs Using Retrieval

Needle 2 indexes every declared tool schema once at initialization using a contrastive embedding head, then retrieves only the top-5 most relevant tools per query based on similarity scoring, keeping inference efficient and token consumption constant regardless of catalog size.

Needle 2, developed by cactus-compute, solves the context window bottleneck for large tool catalogs by implementing a retrieval-based architecture. Instead of loading every tool into the prompt, Needle 2 handles large tool catalogs using retrieval to select only the necessary tools at runtime. This approach allows the framework to scale from a handful of utilities to thousands of tools while maintaining predictable latency.

Embedding and Indexing the Tool Catalog

When you instantiate a Needle object, the framework immediately processes your entire tool collection. In needle/__init__.py, the Needle.__init__ method passes each tool's JSON schema to the native engine via needle_init. The engine contains a built-in contrastive head that embeds every tool schema a single time during this initialization phase.

This one-time embedding operation converts each tool's structured definition into a dense vector representation. By computing these embeddings upfront, Needle 2 avoids the computational overhead of re-processing tool definitions during inference.

Schema Generation Pipeline

The tool schemas themselves are generated from Python callables using utilities in needle/agent/tools.py. This module provides the @tool decorator and functions like build_schema and pydantic_schema that transform decorated functions into the JSON schema format required by the embedding engine.

Query-Driven Retrieval at Runtime

For every user turn, Needle 2 performs a similarity search between the query and the pre-computed tool embeddings. The model first embeds the user query using the same contrastive head, then computes similarity scores against all catalog embeddings.

Only the top-5 scoring tools are selected and injected into the prompt grammar for that specific turn. As documented in doc/apis.md under the "Tool retrieval" section, unselected tools are unreachable for that step—not merely unlikely to be chosen. This hard limit ensures that token consumption remains predictable regardless of catalog size.

Retrieval Outside the Language Model

The retrieval step executes outside the language model. The LLM never sees the full tool catalog, which keeps latency low and prevents context window overflow. This architecture allows Needle 2 to work with hundreds or thousands of tools while presenting only a manageable subset to the model.

Persisting Tool Embeddings with tool_index_path

Computing embeddings for massive catalogs can be expensive. Needle 2 mitigates this through the tool_index_path parameter in Needle.__init__. When provided, the engine stores tool embeddings on disk, keyed by a fingerprint of the tool schemas combined with the model version.

Subsequent runs with identical catalogs and models load embeddings instantly from this cache. If you modify any tool schema, the system detects the change through the fingerprint and re-embeds only the changed entries, as noted in the API documentation.

Why Needle 2 Limits Retrieval to Five Tools

The top-5 limit represents a deliberate design trade-off. This number fits comfortably within the model's context window while still providing sufficient tool diversity for complex tasks. Since the retrieval mechanism filters the catalog before the LLM processes the prompt, the constant token budget per turn remains fixed even as you scale from ten to one thousand tools.

Implementation Example

The following example demonstrates declaring a large catalog and persisting the index:

from needle import Needle, tool

# 1️⃣ Declare a large catalog (more than five tools)

big_catalog = [
    # simple functions decorated with @tool

    *[lambda x, i=i: f"Result {i}" for i in range(20)],   # 20 dummy tools

]

# 2️⃣ Persist embeddings so the first run does the heavy work only once

agent = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",   # ← disk cache for embeddings

)

# 3️⃣ Ask a question that only a few tools are relevant to

response = agent.run(
    "Which tool returns the result for i=7?",
    max_steps=1,
)

print(response["function_calls"])

# → will contain a call only to the tool whose embedding best matches the query

When reusing the index in a new process:

# Re‑using the same index in a new process (instant load)

agent2 = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",   # loads the pre‑computed embeddings

)

print(agent2.run("Give me the result for i=13")["function_calls"])

Summary

• Needle 2 handles large tool catalogs using retrieval by embedding each tool schema once at initialization via the contrastive head in needle_init.
• At runtime, only the top-5 most similar tools enter the context window based on query embedding similarity.
• The tool_index_path parameter enables persistent caching of embeddings, keyed by schema and model fingerprints.
• Tool schemas are generated using the @tool decorator and utilities from needle/agent/tools.py.
• Retrieval occurs outside the LLM, ensuring constant token consumption regardless of catalog size.

Frequently Asked Questions

How does Needle 2 handle more than five tools?

When you declare more than five tools, Needle 2 automatically engages its retrieval system. The engine embeds all tool schemas at initialization, then for each user query, computes similarity scores between the query embedding and tool embeddings. Only the five highest-scoring tools are injected into the prompt; the rest remain unreachable for that turn.

What is the purpose of the tool_index_path parameter?

The tool_index_path parameter specifies a disk location where Needle 2 stores pre-computed tool embeddings. These embeddings are keyed by a fingerprint combining the tool schemas and model version. This allows subsequent initializations to load instantly rather than re-embedding the entire catalog.

Where does the tool retrieval logic execute in Needle 2?

The retrieval logic executes in the native engine during the needle_init call and subsequent query processing, as implemented in needle/__init__.py. This occurs outside the language model itself, meaning the LLM only receives the final filtered set of tools rather than processing the retrieval logic.

How does Needle 2 detect changes in tool schemas?

Needle 2 generates a fingerprint of the tool schemas combined with the model version. When loading from a tool_index_path, it compares the current fingerprint against the cached version. If any schema has changed or the model differs, the system triggers re-embedding of only the modified entries rather than the entire catalog.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}