How to Persist Tool Embeddings Using tool_index_path in Needle

The tool_index_path parameter in Needle enables persistent storage of tool embeddings by writing computed vectors to disk, allowing agents to load pre-calculated representations across sessions without regenerating them.

The open-source Needle framework (cactus-compute/needle) provides a durable indexing mechanism for tool-augmented language models. By configuring the tool_index_path parameter, you create a reusable on-disk database that stores vector representations of your tools, eliminating redundant computation during inference and enabling sharing across distributed worker processes.

How tool_index_path Works

Lazy Loading and Initialization

When initializing an Agent, the framework accepts a tool_index_path argument that defaults to ~/.needle/tool_index.db. The constructor instantiates a ToolIndex object that lazily checks for an existing file at this location. If present, the ToolIndex.load() method deserializes stored embeddings—using either numpy.load or sqlite3 depending on the backend—and re-instantiates the corresponding Tool objects, making them immediately available for model inference.

Embedding Generation and Persistence

For tools not yet present in the index, Needle computes embeddings by passing the tool's description through the model's tokenizer and encoder (internally referenced via model/embed). These vectors are flushed immediately to the specified path, ensuring durability across process restarts. The system uses a write-on-first-use pattern: when a new tool is encountered, its embedding is generated and persisted before the agent executes the tool, guaranteeing that subsequent runs load the vector directly from disk rather than recomputing it.

Cross-Process Sharing

The persistence layer extends beyond single-process boundaries. As implemented in needle/_worker.py, the tool_index_path propagates to all spawned worker subprocesses, allowing multiple agents to share the same on-disk index. This architecture supports high-throughput deployments and multi-node configurations where the index resides on a shared network volume, ensuring consistent tool representations across your compute cluster.

Source Code Implementation

The embedding persistence mechanism spans four critical components in the Needle codebase:

needle/agent/tools.py defines the Tool class and the ToolIndex helper that manages read/write operations. This module contains the serialization logic that converts embedding tensors into SQLite or JSON format, handling both the load() and save operations.

needle/cli.py parses the --toolindexpath command-line flag (the CLI equivalent of tool_index_path) and forwards the value to the Agent constructor, enabling persistent indexes in shell workflows without requiring code changes.

needle/_worker.py receives the tool index path during worker initialization and passes it to subprocesses, ensuring that workers reuse the parent's embedding cache rather than maintaining isolated in-memory copies.

needle/model/run.py integrates the loaded ToolIndex into the inference pipeline, fetching persisted tool embeddings during model execution to inform the model's tool-selection attention mechanism.

Practical Usage Examples

Command-Line Configuration

Specify a custom index location when running Needle from the terminal:

needle run --toolindexpath /data/needle/tool_index.db \
           --model llama-2-7b \
           --prompt "You have access to a calculator tool."

Programmatic Agent Initialization

Create an Agent with a persistent index in Python:

from needle.agent import Agent

# Initialize with a custom tool index path

agent = Agent(
    model_name="llama-2-7b",
    toolindexpath="/var/tmp/needle_tool_index.json"
)

@agent.tool
def add(a: int, b: int) -> int:
    """Return the sum of a and b."""
    return a + b

# Embedding is computed and saved on first use

result = agent.run("What is 12 + 34?")

Reloading Existing Embeddings

Subsequent sessions automatically load persisted embeddings:

from needle.agent import Agent

# Reuse the index created in a previous session

agent = Agent(
    model_name="llama-2-7b",
    toolindexpath="/var/tmp/needle_tool_index.json"
)

# The 'add' tool embedding is loaded from disk automatically

print(agent.run("Use the add tool to compute 7+9."))

Summary

• The tool_index_path parameter controls the on-disk location for tool embeddings, defaulting to ~/.needle/tool_index.db.
• The ToolIndex class in needle/agent/tools.py manages lazy loading and immediate persistence of embedding vectors using SQLite or JSON backends.
• Embeddings are generated once per tool and reused across sessions, eliminating redundant computation during inference.
• Worker processes share the same index file via needle/_worker.py, enabling consistent tool representations in distributed deployments.
• The system writes embeddings immediately after generation when a tool is first used, ensuring the persistent state remains synchronized with tool definitions.

Frequently Asked Questions

What file format does tool_index_path use?

Needle supports both SQLite databases (.db) and plain-text JSON files for the embedding index. The specific backend format depends on your configuration, but both ensure that tool vectors persist across process restarts and system reboots without manual intervention.

Can multiple Agent instances share the same tool_index_path?

Yes. Needle propagates the tool_index_path to all worker subprocesses through needle/_worker.py, allowing multiple agents to read from and write to the same index file concurrently. For distributed deployments, you can mount the path on a network volume to enable multi-node sharing.

What is the default location for the tool index?

When tool_index_path is not specified, the parameter defaults to ~/.needle/tool_index.db in the user's home directory. Needle automatically creates this file the first time a tool requires embedding persistence.

When are new embeddings written to the index?

New embeddings are written immediately after they are computed when a tool is first used. This ensure-on-first-use strategy guarantees that the persistent index contains the most recent embeddings without requiring explicit save calls, while avoiding redundant writes for existing tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}