# How `tool_index_path` Persists Embeddings for Large Tool Catalogues in Needle

> Discover how Needle's tool_index_path efficiently persists embeddings for large tool catalogs using a binary cache, speeding up initialization by loading pre-computed data.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-09-10

---

**The `tool_index_path` parameter in Needle enables persistent storage of tool embeddings via a fingerprint-based binary cache, allowing the framework to load pre-computed embeddings instantly rather than regenerating them on every initialization.**

When working with extensive tool catalogues in the `cactus-compute/needle` repository, the embedding computation cost for retrieval-based tool selection becomes prohibitive without caching. The optional `tool_index_path` argument solves this by providing a durable on-disk index that maps deterministic schema fingerprints to their corresponding embedding vectors, eliminating redundant computation across process restarts.

## The Retrieval Mode Threshold

Needle operates in two distinct modes depending on the size of your tool catalogue. When you instantiate a `Needle` agent with **more than five declared tools**, the framework switches from inline schema rendering to a retrieval-based approach. In this mode, each tool's JSON schema is processed by a built-in contrastive head to generate embeddings. Only the five highest-scoring tool embeddings are injected into the model's context window for each turn, significantly reducing context window usage for large catalogues.

## Fingerprint-Based Embedding Persistence

The `tool_index_path` mechanism relies on deterministic fingerprinting to ensure cache validity and enable incremental updates.

### Embedding Generation and Fingerprinting

At initialization, the native C library (`needle_init`) computes a deterministic fingerprint by hashing the concatenated tool schemas together with the model generation configuration. This occurs in `Needle.__init__` within [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 153–176). The fingerprint acts as a unique identifier for the exact state of your tool catalogue and model combination.

### Persisting the Index to Disk

If the file specified by `tool_index_path` does not exist, Needle writes a binary index file containing a flat table of `(fingerprint → embedding)` entries. According to the source code in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the path string is UTF-8 encoded and stored in `self._tool_index_path` before being passed to the native initialization routine. The file format is opaque to Python but efficiently stores the embedding vectors for later retrieval.

### Loading Cached Embeddings

On subsequent invocations, passing the same path to `needle_init` triggers a fingerprint comparison. If the stored fingerprint matches the current schemas and model version, the embeddings are loaded directly from disk instead of being recomputed. This makes startup virtually instantaneous even for catalogues containing thousands of tools. When using fine-tuned models, the path is forwarded to the `FineTuneWorker` constructor via `os.fspath(tool_index_path)` to ensure the worker locates the correct cache file.

### Incremental Updates

When a tool schema changes, only the altered entry generates a different fingerprint. The library detects this mismatch and recomputes the embedding for that specific tool only, appending or updating the entry in the binary index while leaving previously cached embeddings untouched. This incremental approach ensures that minor catalogue modifications do not invalidate the entire cache.

## Implementation Details

Under the hood, the persistence mechanism relies on Python-to-C interoperability. The `Needle._bind` method (lines 190–199 in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)) passes the tool index path to the native library using `ctypes.c_char_p` encoding:

```python
lib.needle_init(self._system, self._tools_json, self._tool_index_path)

```

The native `needle_init` function handles the low-level file I/O operations, checking for existing indices at the provided path and managing the binary serialization format. Documentation in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) (lines 165–168) describes this retrieval mechanism and persistence behavior, while [`llms.txt`](https://github.com/cactus-compute/needle/blob/main/llms.txt) (line 22) lists `tool_index_path` as the recommended configuration option for large tool catalogues.

## Practical Usage Example

The following example demonstrates how to leverage `tool_index_path` for a large tool catalogue:

```python
import needle

# Declare a large catalogue of tools (e.g., dozens of functions)

big_catalogue = [my_tool_1, my_tool_2, my_tool_3, my_tool_4, my_tool_5, my_tool_6]

# Provide a path where embeddings will be persisted

agent = needle.Needle(
    tools=big_catalogue,
    tool_index_path="tools.idx"   # ← persisted on disk

)

# First run – embeddings are computed and saved

response = agent.run("What can I do with the kitchen appliances?")
print(response["function_calls"])

# Subsequent run – embeddings are loaded instantly from tools.idx

agent2 = needle.Needle(
    tools=big_catalogue,
    tool_index_path="tools.idx"
)
response2 = agent2.run("Turn on the coffee maker")
print(response2["function_calls"])

```

## Summary

- **`tool_index_path`** provides a persistent binary cache for tool embeddings, stored as a flat table mapping fingerprints to vectors.
- Needle automatically switches to **retrieval mode** when declaring more than five tools, embedding schemas via a built-in contrastive head.
- **Deterministic fingerprinting** combines tool schemas and model generation parameters to ensure cache validity across sessions.
- The native `needle_init` function handles **incremental updates**, recomputing embeddings only for modified tools while preserving cached entries.
- The implementation passes paths via **ctypes** (`ctypes.c_char_p`) and supports fine-tuned models through the `FineTuneWorker` class.

## Frequently Asked Questions

### What file format does tool_index_path use?

The `tool_index_path` file uses a proprietary binary format that stores a flat table of fingerprint-to-embedding mappings. While this format is opaque to Python and managed entirely by the native C library, it functions essentially as a key-value store where keys are deterministic hashes of your tool schemas and values are the corresponding embedding vectors.

### How does Needle handle tool schema changes with an existing index?

When tool schemas change, Needle recomputes the fingerprint for only the modified tools during initialization. The native library detects fingerprint mismatches between the current schemas and the cached index, regenerates embeddings for the altered entries, and updates the binary file at `tool_index_path` while preserving all unchanged embeddings. This incremental update strategy prevents full cache invalidation.

### Can tool_index_path be shared across different machines?

Sharing a `tool_index_path` file between different machines is possible only if the tool catalogues, model versions, and Needle library versions are identical, since the fingerprint incorporates these parameters. However, the binary file is platform-specific due to the native C implementation, so portability across different architectures or operating systems is not guaranteed without revalidation.

### What happens if I don't provide tool_index_path?

If `tool_index_path` is omitted, Needle computes embeddings in memory during initialization but does not persist them to disk. For catalogues exceeding five tools, this means paying the embedding computation cost on every process restart, which becomes prohibitively expensive for large catalogues containing hundreds or thousands of tool definitions.