How `tool_index_path` Persists Embeddings for Large Tool Catalogues in Needle

The tool_index_path parameter in Needle enables persistent storage of tool embeddings via a fingerprint-based binary cache, allowing the framework to load pre-computed embeddings instantly rather than regenerating them on every initialization.

When working with extensive tool catalogues in the cactus-compute/needle repository, the embedding computation cost for retrieval-based tool selection becomes prohibitive without caching. The optional tool_index_path argument solves this by providing a durable on-disk index that maps deterministic schema fingerprints to their corresponding embedding vectors, eliminating redundant computation across process restarts.

The Retrieval Mode Threshold

Needle operates in two distinct modes depending on the size of your tool catalogue. When you instantiate a Needle agent with more than five declared tools, the framework switches from inline schema rendering to a retrieval-based approach. In this mode, each tool's JSON schema is processed by a built-in contrastive head to generate embeddings. Only the five highest-scoring tool embeddings are injected into the model's context window for each turn, significantly reducing context window usage for large catalogues.

Fingerprint-Based Embedding Persistence

The tool_index_path mechanism relies on deterministic fingerprinting to ensure cache validity and enable incremental updates.

Embedding Generation and Fingerprinting

At initialization, the native C library (needle_init) computes a deterministic fingerprint by hashing the concatenated tool schemas together with the model generation configuration. This occurs in Needle.__init__ within needle/__init__.py (lines 153–176). The fingerprint acts as a unique identifier for the exact state of your tool catalogue and model combination.

Persisting the Index to Disk

If the file specified by tool_index_path does not exist, Needle writes a binary index file containing a flat table of (fingerprint → embedding) entries. According to the source code in needle/__init__.py, the path string is UTF-8 encoded and stored in self._tool_index_path before being passed to the native initialization routine. The file format is opaque to Python but efficiently stores the embedding vectors for later retrieval.

Loading Cached Embeddings

On subsequent invocations, passing the same path to needle_init triggers a fingerprint comparison. If the stored fingerprint matches the current schemas and model version, the embeddings are loaded directly from disk instead of being recomputed. This makes startup virtually instantaneous even for catalogues containing thousands of tools. When using fine-tuned models, the path is forwarded to the FineTuneWorker constructor via os.fspath(tool_index_path) to ensure the worker locates the correct cache file.

Incremental Updates

When a tool schema changes, only the altered entry generates a different fingerprint. The library detects this mismatch and recomputes the embedding for that specific tool only, appending or updating the entry in the binary index while leaving previously cached embeddings untouched. This incremental approach ensures that minor catalogue modifications do not invalidate the entire cache.

Implementation Details

Under the hood, the persistence mechanism relies on Python-to-C interoperability. The Needle._bind method (lines 190–199 in needle/__init__.py) passes the tool index path to the native library using ctypes.c_char_p encoding:

lib.needle_init(self._system, self._tools_json, self._tool_index_path)

The native needle_init function handles the low-level file I/O operations, checking for existing indices at the provided path and managing the binary serialization format. Documentation in doc/apis.md (lines 165–168) describes this retrieval mechanism and persistence behavior, while llms.txt (line 22) lists tool_index_path as the recommended configuration option for large tool catalogues.

Practical Usage Example

The following example demonstrates how to leverage tool_index_path for a large tool catalogue:

import needle

# Declare a large catalogue of tools (e.g., dozens of functions)

big_catalogue = [my_tool_1, my_tool_2, my_tool_3, my_tool_4, my_tool_5, my_tool_6]

# Provide a path where embeddings will be persisted

agent = needle.Needle(
    tools=big_catalogue,
    tool_index_path="tools.idx"   # ← persisted on disk

)

# First run – embeddings are computed and saved

response = agent.run("What can I do with the kitchen appliances?")
print(response["function_calls"])

# Subsequent run – embeddings are loaded instantly from tools.idx

agent2 = needle.Needle(
    tools=big_catalogue,
    tool_index_path="tools.idx"
)
response2 = agent2.run("Turn on the coffee maker")
print(response2["function_calls"])

Summary

• tool_index_path provides a persistent binary cache for tool embeddings, stored as a flat table mapping fingerprints to vectors.
• Needle automatically switches to retrieval mode when declaring more than five tools, embedding schemas via a built-in contrastive head.
• Deterministic fingerprinting combines tool schemas and model generation parameters to ensure cache validity across sessions.
• The native needle_init function handles incremental updates, recomputing embeddings only for modified tools while preserving cached entries.
• The implementation passes paths via ctypes (ctypes.c_char_p) and supports fine-tuned models through the FineTuneWorker class.

Frequently Asked Questions

What file format does tool_index_path use?

The tool_index_path file uses a proprietary binary format that stores a flat table of fingerprint-to-embedding mappings. While this format is opaque to Python and managed entirely by the native C library, it functions essentially as a key-value store where keys are deterministic hashes of your tool schemas and values are the corresponding embedding vectors.

How does Needle handle tool schema changes with an existing index?

When tool schemas change, Needle recomputes the fingerprint for only the modified tools during initialization. The native library detects fingerprint mismatches between the current schemas and the cached index, regenerates embeddings for the altered entries, and updates the binary file at tool_index_path while preserving all unchanged embeddings. This incremental update strategy prevents full cache invalidation.

Can tool_index_path be shared across different machines?

Sharing a tool_index_path file between different machines is possible only if the tool catalogues, model versions, and Needle library versions are identical, since the fingerprint incorporates these parameters. However, the binary file is platform-specific due to the native C implementation, so portability across different architectures or operating systems is not guaranteed without revalidation.

What happens if I don't provide tool_index_path?

If tool_index_path is omitted, Needle computes embeddings in memory during initialization but does not persist them to disk. For catalogues exceeding five tools, this means paying the embedding computation cost on every process restart, which becomes prohibitively expensive for large catalogues containing hundreds or thousands of tool definitions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}