How to Persist Tool Embeddings for Faster Re‑initialization in Needle 2

You can persist tool embeddings in Needle 2 by using the ToolEmbedding.save() and ToolEmbedding.load() methods, which store NumPy arrays to disk and reload them on subsequent runs to bypass expensive re‑creation.

Needle 2 generates embeddings for its tool‑use functions at startup, and rebuilding these vectors every session adds noticeable latency—especially on slower hardware. The cactus‑compute/needle repository includes a built‑in caching mechanism that eliminates this overhead by persisting embeddings to disk. This article walks through the exact implementation paths, environment configuration, and code patterns to enable fast re‑initialization.

How Tool Embedding Persistence Works in Needle 2

The persistence system operates across five coordinated steps in the source code. Each step maps to specific file paths and method implementations.

Step 1: Cache Directory Resolution

When Needle is first imported, it resolves where to store embeddings. By default, this is ~/.cache/needle.

In needle/agent/fetch.py (lines 67‑71), the library creates this directory if missing:

# Excerpt from needle/agent/fetch.py

CACHE_DIR = Path.home() / ".cache" / "needle"
CACHE_DIR.mkdir(parents=True, exist_ok=True)

Step 2: Embedding Generation

The first time the LLM calls a tool, the ToolEmbedding class constructs a dense vector from the tool's name, description, and JSON schema.

In needle/agent/tools.py (lines 18‑30), this construction parses schemas and tokenizes descriptions—operations that dominate startup time.

Step 3: Persist Vectors to Disk

After successful generation, ToolEmbedding.save() writes the NumPy array to tool_embeddings.npy inside the cache directory.

From needle/agent/tools.py (lines 140‑152):

def save(self) -> None:
    """Persist embeddings to disk as float32 NumPy array."""
    path = self._cache_path / "tool_embeddings.npy"
    np.save(path, self.vectors.astype(np.float32))

Step 4: Load on Subsequent Startup

On later launches, ToolEmbedding.load() checks for the cached file and short‑circuits reconstruction.

In needle/agent/tools.py (lines 156‑165):

def load(self) -> bool:
    """Load embeddings from disk if available."""
    path = self._cache_path / "tool_embeddings.npy"
    if path.exists():
        self.vectors = np.load(path)
        return True
    return False

Step 5: Override Cache Location with Environment Variable

Both save() and load() respect the NEEDLE_EMBEDDINGS_DIR environment variable for custom paths.

In needle/agent/fetch.py (lines 23‑30), the resolution logic:

EMBEDDINGS_DIR = os.environ.get(
    "NEEDLE_EMBEDDINGS_DIR",
    str(Path.home() / ".cache" / "needle")
)

Practical Implementation: Enable Embedding Persistence

The needle/cli.py entry point automatically triggers ToolEmbedding.load() before any LLM interaction. For custom scripts or the Python API, explicitly use the methods below.

Basic Workflow

from needle.agent.tools import ToolEmbedding

# Automatically loads cached embeddings if available

embedding = ToolEmbedding()
embedding.load()

# If no cache existed, first tool call builds and auto-saves

# Subsequent runs load instantly from disk

Custom Cache Location

import os

os.environ["NEEDLE_EMBEDDINGS_DIR"] = "/my/custom/cache"

from needle.agent.tools import ToolEmbedding

embedding = ToolEmbedding()
found_cache = embedding.load()  # True if loaded from custom path

Force Cache Refresh

After adding new tools or modifying schemas, invalidate the cache:

from needle.agent.tools import ToolEmbedding

embedding = ToolEmbedding()

# Method 1: Use the built-in clear method

embedding.clear_cache()   # Removes tool_embeddings.npy

embedding.load()          # Rebuilds on next tool usage

# Method 2: Manual file deletion

import os
cache_file = os.path.join(os.environ.get("NEEDLE_EMBEDDINGS_DIR", "~/.cache/needle"), "tool_embeddings.npy")
os.remove(os.path.expanduser(cache_file))

Performance Characteristics

Aspect	Detail
Storage format	NumPy .npy file with float32 arrays
Default location	~/.cache/needle/tool_embeddings.npy
Override mechanism	NEEDLE_EMBEDDINGS_DIR environment variable
Typical speedup	Eliminates schema parsing and tokenization on every startup
Portability	.npy files are cross‑platform and version‑stable

The choice of NumPy's binary format ensures fast I/O without serialization overhead, and float32 precision balances embedding quality with storage efficiency.

Source File Reference

File	Responsibility
needle/agent/fetch.py	Cache path resolution; NEEDLE_EMBEDDINGS_DIR handling
needle/agent/tools.py	ToolEmbedding class with save(), load(), clear_cache()
needle/cli.py	Automatic ToolEmbedding.load() invocation at startup

Summary

• Needle 2 persists tool embeddings via ToolEmbedding.save() and ToolEmbedding.load() in needle/agent/tools.py
• The default cache directory is ~/.cache/needle, configurable through NEEDLE_EMBEDDINGS_DIR
• Embeddings are stored as NumPy float32 arrays in tool_embeddings.npy for fast, portable I/O
• Use clear_cache() or delete the .npy file to force regeneration after tool changes
• The CLI entry point auto‑loads cached embeddings, requiring no code changes for standard usage

Frequently Asked Questions

What file format does Needle 2 use for persisted embeddings?

Needle 2 uses NumPy's .npy format with float32 precision. This provides fast binary I/O without additional dependencies and maintains compatibility across Python versions and operating systems. The implementation in needle/agent/tools.py:140‑152 calls np.save() and np.load() directly.

Can I change where tool embeddings are stored?

Yes. Set the NEEDLE_EMBEDDINGS_DIR environment variable before importing Needle. The resolution logic in needle/agent/fetch.py:23‑30 checks this variable first, falling back to ~/.cache/needle only when unset. Both save() and load() respect the overridden path.

Why are my tool embeddings being rebuilt every session?

This occurs when the cache file cannot be found or loaded. Verify that NEEDLE_EMBEDDINGS_DIR points to a writable, persistent directory. Ensure the first tool call completes successfully—only then does ToolEmbedding.save() trigger. Check needle/agent/tools.py:156‑165 for the existence check logic.

How do I force re‑generation of embeddings after adding new tools?

Call ToolEmbedding.clear_cache() to remove the persisted file, then invoke ToolEmbedding.load() to trigger reconstruction during the next tool usage. Alternatively, manually delete tool_embeddings.npy from your cache directory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}