How to Handle Large Tool Catalogs with Needle 2's Retrieval Mechanism

Needle 2 indexes every tool schema once at initialization using a native contrastive embedding head, then retrieves only the top-5 most relevant tools per query to maintain efficient inference with hundreds or thousands of tools.

Managing extensive tool libraries in LLM applications typically risks exceeding context windows and degrading performance. The cactus-compute/needle repository solves this through a built-in retrieval system that embeds tool schemas separately from the language model, enabling scalable agent architectures without token bloat.

Understanding the Retrieval Architecture

Needle 2 employs a two-phase approach to handle large catalogs: pre-computation of tool embeddings followed by similarity-based retrieval at inference time.

One-Time Schema Embedding

When you instantiate a Needle object, the native engine processes your entire tool catalog immediately. In needle/__init__.py, the Needle.__init__ method (lines 54–66) passes each tool's JSON schema to the underlying engine via needle_init. The engine contains a built-in contrastive head that embeds every tool schema a single time, regardless of how many times those tools will be used during the session.

This initialization happens outside the LLM context, meaning the language model itself never processes the full catalog during inference.

Query-Driven Top-5 Selection

For every user turn, Needle 2 performs three operations:

1. Embeds the user query using the same contrastive head
2. Computes similarity scores between the query embedding and all pre-computed tool embeddings
3. Selects only the top-5 scoring tools to inject into the prompt grammar

As documented in doc/apis.md under the "Tool retrieval" section, only these five highest-scoring tools enter the context for that specific turn. Unselected tools are unreachable, not merely unlikely to be chosen. This hard limit ensures predictable token consumption and latency regardless of catalog size.

Persisting Tool Embeddings for Performance

Computing embeddings for massive catalogs can be computationally expensive. Needle 2 provides the tool_index_path parameter in Needle.__init__ to cache these embeddings between sessions.

When you supply a file path to this argument, the engine stores embeddings on disk, keyed by a cryptographic fingerprint of:

• The tool schemas
• The model version

On subsequent runs, if the fingerprint matches, the embeddings load instantly. If you modify any tool schema, only the changed entries trigger re-embedding, while the rest load from cache. This incremental update behavior makes it practical to work with catalogs containing thousands of tools without paying the initialization penalty on every startup.

Practical Implementation

The following example demonstrates declaring a large catalog, persisting embeddings, and reusing the index:

from needle import Needle

# Declare a large catalog with more than five tools

big_catalog = [
    lambda x, i=i: f"Result {i}" for i in range(20)  # 20 dummy tools

]

# Persist embeddings to disk so initialization cost is paid only once

agent = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",  # Disk cache for embeddings

)

# Query targeting specific tools

response = agent.run(
    "Which tool returns the result for i=7?",
    max_steps=1,
)
print(response["function_calls"])

# Output contains only the tool whose embedding matches the query

To reuse the pre-computed index in a new process:

# Loads pre-computed embeddings instantly via fingerprint matching

agent2 = Needle(
    tools=big_catalog,
    tool_index_path="my_tools.idx",
)
result = agent2.run("Give me the result for i=13")

The @tool decorator and schema builders reside in needle/agent/tools.py, which converts Python callables into the JSON schemas required by the embedding engine.

Why Five Tools? Design Trade-offs

The retrieval mechanism limits selection to five tools as a deliberate architectural constraint. This number balances two competing requirements:

• Context efficiency: Five tool schemas fit comfortably within the model's context window alongside conversation history and system prompts
• Sufficient variety: Five options provide enough semantic diversity for the model to select appropriate functionality

Because retrieval occurs outside the language model, the LLM never sees the full catalog, ensuring that inference latency remains constant whether you declare 10 tools or 1,000. The similarity computation happens in the native engine, not in Python, maintaining performance even with high-dimensional embedding comparisons.

Summary

• Needle 2 handles large tool catalogs by embedding schemas once at initialization using a native contrastive head in needle/__init__.py
• At inference time, only the top-5 most relevant tools enter the prompt context based on query similarity scoring
• Use the tool_index_path parameter to persist embeddings to disk, keyed by schema and model fingerprints for instant reloading
• Unselected tools are unreachable during a given turn, not just deprioritized, ensuring strict context window management
• The @tool decorator in needle/agent/tools.py handles schema generation from Python functions

Frequently Asked Questions

How does Needle 2 handle tool catalogs with thousands of entries?

Needle 2 scales to thousands of tools by indexing them outside the LLM context. The native engine embeds all tool schemas during Needle initialization, then retrieves only the top five matches per query. This prevents the context window from growing with catalog size, keeping token consumption and inference latency constant regardless of whether you have 50 or 5,000 tools.

What is the tool_index_path parameter in Needle 2?

The tool_index_path parameter in Needle.__init__ specifies a file path where the engine stores computed tool embeddings. The cache uses a fingerprint of the tool schemas and model version as a key. If the fingerprint matches on subsequent runs, embeddings load instantly; if schemas changed, only the modified tools get re-embedded. This dramatically reduces startup time for large catalogs.

Why does Needle 2 only retrieve five tools per turn?

Five represents a fixed trade-off between providing sufficient tool variety for the model to choose from and maintaining strict context limits. According to the implementation in doc/apis.md, this number fits comfortably within the model's available context while ensuring that unselected tools are completely unreachable during that turn, preventing token waste on irrelevant schemas.

Where is the tool retrieval logic implemented in the Needle codebase?

The retrieval orchestration begins in needle/__init__.py where Needle.__init__ passes schemas to the native engine via needle_init. The actual embedding and similarity computation occur in the native layer's contrastive head. Documentation describing the "top-5" rule and retrieval behavior lives in doc/apis.md, while needle/agent/tools.py handles the Python-side schema generation and the @tool decorator.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}