How to Persist Tool Embeddings for Fast Reloads with Large Tool Catalogues in Needle

Persist tool embeddings on disk using Needle's standard cache directory (~/.cache/cactus-needle/tool_embeddings/) so they load instantly on subsequent runs instead of recomputing for every inference pass.

Needle exposes Python functions as LLM tools via the @tool decorator, generating JSON schemas through build_schema(). When your agent manages hundreds of tools, the embedding step becomes a bottleneck—each tool's name and description must be vectorized so the model can rank relevance. This guide shows how to cache these embeddings using patterns already established in the Needle codebase.

Why Tool Embedding Caching Matters

Every inference pass with tool selection requires embedding all available tool descriptions. For large catalogues, this redundant computation adds seconds to startup time. The Needle repository already solves similar problems with two caching strategies:

• In-memory caches in needle/model/run.py—_decode_fn_cache and _rollout_cache use functools.lru_cache for per-process persistence
• On-disk caches in ~/.cache/cactus-needle/—established in needle/__init__.py and used for compiled native libraries and Hugging Face model files

Tool embeddings deserve the same on-disk treatment: compute once, persist, reload instantly.

Implementing Persistent Tool Embeddings

Step 1: Hash the Tool Catalogue for Version Safety

Stale embeddings must never load against a changed schema. Compute a deterministic hash of the tool list so any schema modification invalidates the cache:

import json
import hashlib


def _catalogue_hash(tools):
    """Deterministic hash—changes when any tool schema changes."""
    canon = json.dumps(sorted(tools, key=lambda t: t["name"]), separators=(",", ":"))
    return hashlib.sha256(canon.encode()).hexdigest()[:12]

Step 2: Define the Cache Location

Follow Needle's convention from needle/__init__.py (lines 66-71) and needle/cli.py (lines 242-258):

import os
from pathlib import Path


CACHE_ROOT = Path(os.path.expanduser("~")) / ".cache" / "cactus-needle" / "tool_embeddings"


def _cache_path(model_id: str, catalogue_hash: str) -> Path:
    """Path where embeddings for a given model/catalogue live."""
    return CACHE_ROOT / f"{model_id}_{catalogue_hash}.npz"

Step 3: Compute and Persist Embeddings

Reuse the same embedding routine that needle/_worker.py employs for standard text—typically wrapping the native needle_embed call. Store vectors with tool names for fast lookup:

import numpy as np
from .agent.tools import build_schema
from .model.run import embed_text  # Internal embedding function


def compute_and_save(model_id: str, tools):
    """Compute embeddings for every tool and persist them."""
    CACHE_ROOT.mkdir(parents=True, exist_ok=True)

    tool_texts = [
        f"{t['name']}: {t.get('description', '')}"
        for t in tools
    ]

    # Same routine used for plain text embeddings in _worker.py lines 169-179

    vectors = np.stack([embed_text(txt) for txt in tool_texts])

    catalogue_hash = _catalogue_hash(tools)
    path = _cache_path(model_id, catalogue_hash)
    np.savez_compressed(path, names=[t["name"] for t in tools], vectors=vectors)
    return path

Step 4: Fast Reload with Cache Miss Handling

def load_embeddings(model_id: str, tools):
    """Load embeddings if they exist, otherwise compute and cache them."""
    catalogue_hash = _catalogue_hash(tools)
    path = _cache_path(model_id, catalogue_hash)

    if path.is_file():
        data = np.load(path, allow_pickle=False)
        return dict(zip(data["names"], data["vectors"]))

    compute_and_save(model_id, tools)
    return load_embeddings(model_id, tools)

Integrating with the Worker Pipeline

Hook the cache loader into needle/_worker.py where tool schemas are prepared:

from .agent.tools import build_schema
from .utils.tool_embeddings import load_embeddings


def _prepare_tool_embeddings(model_id: str, tool_fns):
    """Create fast lookup table: tool-name → embedding."""
    tool_schemas = [build_schema(fn) for fn in tool_fns]
    return load_embeddings(model_id, tool_schemas)

This replaces repeated embedding computation with a single dictionary lookup after the first run.

Key Files and Their Roles

File	Purpose	Critical Lines
needle/agent/tools.py	@tool decorator and build_schema()—generates JSON schemas that get embedded	Tool definitions
needle/_worker.py	Native embedding calls via needle_embed; integration point for cached tool vectors	Lines 169-179
needle/model/run.py	In-memory caching patterns (_decode_fn_cache, _rollout_cache)	Caching reference
needle/__init__.py	Cache directory setup: ~/.cache/cactus-needle	Lines 66-71
needle/cli.py	On-disk cache operations for model files	Lines 242-258

Performance Characteristics

• First run: Embedding computation scales linearly with catalogue size (milliseconds per tool)
• Subsequent runs: Loading from .npz via np.load—typically sub-100ms even for thousands of tools
• Invalidation: Automatic on any schema change via catalogue hash; manual cleanup by deleting ~/.cache/cactus-needle/tool_embeddings/

Summary

• Hash the catalogue to version-cache entries and prevent stale embeddings
• Use Needle's standard cache path (~/.cache/cactus-needle/tool_embeddings/) for consistency with existing on-disk storage
• Reuse internal embedding routines from _worker.py rather than duplicating vectorization logic
• Store as compressed NumPy archives for fast I/O and memory mapping
• Integrate at schema preparation time in the worker pipeline for transparent fast reloads

Frequently Asked Questions

How does the cache know when tools have changed?

The _catalogue_hash() function computes a SHA256 hash of the canonicalized JSON representation of all tool schemas. Any modification to names, descriptions, or parameters changes the hash, forcing re-computation on next load.

Can I use this with multiple models simultaneously?

Yes—the cache path includes model_id in the filename, so embeddings for different models remain isolated. Each model loads only its compatible vectors.

What if I need to clear the cache manually?

Delete the directory ~/.cache/cactus-needle/tool_embeddings/ or specific .npz files. The code gracefully handles missing cache files by recomputing embeddings.

Does this work with dynamically added tools at runtime?

If tools are added after initial load, the catalogue hash changes, triggering cache invalidation. For truly dynamic scenarios, consider a hybrid approach: load cached embeddings for the base catalogue, compute embeddings for new tools on-demand, and periodically merge back to disk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}