How to Persist Tool Embeddings Across Runs Using `tool_index_path` in Needle

Set the tool_index_path parameter when creating a Needle agent to cache tool embeddings to disk, enabling instant loading in subsequent runs without recomputing embeddings.

The Needle inference engine uses a lightweight contrastive head to embed tool schemas for retrieval-based tool selection. When you have more than five tools, Needle performs a retrieval step to select the most relevant tools for each turn. Recomputing these embeddings on every startup creates significant latency for large tool catalogs. The tool_index_path parameter solves this by persisting embeddings to disk with automatic fingerprint-based versioning.

How tool_index_path Works in the Needle Engine

Needle's Python wrapper passes the tool_index_path through to the native engine's initialization routine. Understanding this flow helps you use the feature effectively.

Constructor Storage and Encoding

In needle/__init__.py (lines 56-72), the Needle class constructor accepts tool_index_path as a string and stores it as a UTF-8-encoded byte sequence:

# needle/__init__.py – constructor handling

self._tool_index_path = tool_index_path.encode("utf-8") if tool_index_path else b""

An empty string disables persistence entirely, forcing the engine to use in-memory embeddings only.

Native Engine Initialization

The wrapper forwards the encoded path to needle_init (lines 95-99):

# needle/__init__.py – engine binding

self._engine = needle_init(
    system,
    json.dumps([t._tool_schema for t in tools]),
    self._tool_index_path,  # persisted index location

)

The native engine performs three operations based on the path:

1. Check: Verify if a file exists at the specified path
2. Load or Compute: Load existing embeddings if fingerprint matches; otherwise compute and store new embeddings
3. Fingerprint Matching: Index keys combine tool schema hashes with model version for selective updates

Fingerprint-Based Selective Updates

The index uses a fingerprint derived from both the tool schema and the model version. This design enables partial re-embedding when catalogs evolve.

Scenario	Behavior
Same schemas, same model	Instant load from disk
Modified tool schema	Re-embed only changed tools
Model version change	Re-embed all tools with new model

As documented in doc/apis.md (lines 146-150): "tool_index_path persists the embeddings on disk, keyed by a fingerprint … a matching fingerprint loads instantly, a changed schema re‑embeds only what changed."

Practical Implementation Example

The following example demonstrates persistence across multiple agent instantiations:

import needle

# Declare a large tool catalog (>5 tools triggers retrieval)

tools = []
for i in range(20):
    @needle.tool
    def make_echo(i=i):
        def echo(msg: str) -> dict:
            """Return the message unchanged."""
            return {"msg": msg}
        return echo
    tools.append(make_echo())

# First run: embeds all tools and writes index to disk

agent = needle.Needle(
    tools=tools,
    tool_index_path="my_tool_index.idx",  # creates on first run

)
response = agent.run("repeat 'hello' using any tool")

# Subsequent runs: loads embeddings instantly from disk

agent2 = needle.Needle(
    tools=tools,
    tool_index_path="my_tool_index.idx",  # reuses existing index

)
response2 = agent2.run("repeat 'world'")

Performance Characteristics

Metric	Without Persistence	With tool_index_path
First startup	Embedding latency	Embedding latency + disk write
Restarts	Full re-embedding	Near-instant load
Cross-process sharing	Not possible	Shared via filesystem
Schema updates	N/A (always recomputed)	Incremental update only

File Location and Permissions

The tool_index_path accepts any writeable filesystem location. Relative paths resolve against the working directory. Consider these conventions:

• Development: ./.needle/tool_index.idx (project-local, gitignored)
• Production: /var/cache/needle/tool_index.idx or container volume mount
• Containerized: Ensure the path resides on a persistent volume

Key Source Files

Understanding the implementation requires examining these files:

• needle/__init__.py – Python wrapper storing _tool_index_path and binding to native needle_init
• doc/apis.md – API documentation covering persistence semantics
• needle/agent/tools.py – Tool decorator and schema generation used before fingerprinting
• tests/test_tools.py – Unit tests validating tool handling behavior

Summary

• tool_index_path enables disk-based persistence of expensive tool embeddings
• The index is fingerprint-keyed by schema and model version for selective updates
• Set to a writable file path to enable persistence; empty string disables it
• Subsequent runs load embeddings instantly, dramatically reducing startup time
• Schema or model changes trigger partial re-embedding automatically

Frequently Asked Questions

What happens if I change a tool's docstring but not its signature?

The fingerprint includes the complete schema, which incorporates the docstring for embedding purposes. Modifying the docstring changes the fingerprint, triggering re-embedding for that specific tool. The engine preserves embeddings for all unchanged tools.

Can multiple processes share the same tool_index_path file simultaneously?

The underlying native engine handles concurrent access through appropriate file locking. Multiple processes can reference the same path safely; however, simultaneous writes (from processes with different schemas) may cause contention. For production deployments with frequent schema changes, use process-specific paths or implement external coordination.

Does tool_index_path work with cloud storage or network filesystems?

The native engine performs standard file I/O operations. Network filesystems (NFS, EFS) and cloud-mounted volumes (FUSE-based GCS/S3 mounts) function correctly provided they support standard POSIX file semantics. Latency-sensitive applications benefit from local SSD storage for index files.

How do I invalidate or rebuild the index manually?

Delete the file specified by tool_index_path. The engine will detect the missing file on next initialization, compute fresh embeddings, and write a new index. There is no built-in API for explicit invalidation—filesystem deletion is the intended mechanism.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}