# How to Persist Tool Embeddings for Fast Reloads with Large Tool Catalogues in Needle

> Speed up tool embedding reloads in Needle with large catalogues. Persist embeddings to disk for instant loading and faster inference. Learn how with Cactus Compute.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-06

---

**Persist tool embeddings on disk using Needle's standard cache directory (`~/.cache/cactus-needle/tool_embeddings/`) so they load instantly on subsequent runs instead of recomputing for every inference pass.**

Needle exposes Python functions as LLM tools via the `@tool` decorator, generating JSON schemas through `build_schema()`. When your agent manages hundreds of tools, the embedding step becomes a bottleneck—each tool's name and description must be vectorized so the model can rank relevance. This guide shows how to cache these embeddings using patterns already established in the Needle codebase.

## Why Tool Embedding Caching Matters

Every inference pass with tool selection requires embedding all available tool descriptions. For large catalogues, this redundant computation adds seconds to startup time. The Needle repository already solves similar problems with two caching strategies:

- **In-memory caches** in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)—`_decode_fn_cache` and `_rollout_cache` use `functools.lru_cache` for per-process persistence
- **On-disk caches** in `~/.cache/cactus-needle/`—established in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) and used for compiled native libraries and Hugging Face model files

Tool embeddings deserve the same on-disk treatment: compute once, persist, reload instantly.

## Implementing Persistent Tool Embeddings

### Step 1: Hash the Tool Catalogue for Version Safety

Stale embeddings must never load against a changed schema. Compute a deterministic hash of the tool list so any schema modification invalidates the cache:

```python
import json
import hashlib


def _catalogue_hash(tools):
    """Deterministic hash—changes when any tool schema changes."""
    canon = json.dumps(sorted(tools, key=lambda t: t["name"]), separators=(",", ":"))
    return hashlib.sha256(canon.encode()).hexdigest()[:12]

```

### Step 2: Define the Cache Location

Follow Needle's convention from [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 66-71) and [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) (lines 242-258):

```python
import os
from pathlib import Path


CACHE_ROOT = Path(os.path.expanduser("~")) / ".cache" / "cactus-needle" / "tool_embeddings"


def _cache_path(model_id: str, catalogue_hash: str) -> Path:
    """Path where embeddings for a given model/catalogue live."""
    return CACHE_ROOT / f"{model_id}_{catalogue_hash}.npz"

```

### Step 3: Compute and Persist Embeddings

Reuse the same embedding routine that [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) employs for standard text—typically wrapping the native `needle_embed` call. Store vectors with tool names for fast lookup:

```python
import numpy as np
from .agent.tools import build_schema
from .model.run import embed_text  # Internal embedding function


def compute_and_save(model_id: str, tools):
    """Compute embeddings for every tool and persist them."""
    CACHE_ROOT.mkdir(parents=True, exist_ok=True)

    tool_texts = [
        f"{t['name']}: {t.get('description', '')}"
        for t in tools
    ]

    # Same routine used for plain text embeddings in _worker.py lines 169-179

    vectors = np.stack([embed_text(txt) for txt in tool_texts])

    catalogue_hash = _catalogue_hash(tools)
    path = _cache_path(model_id, catalogue_hash)
    np.savez_compressed(path, names=[t["name"] for t in tools], vectors=vectors)
    return path

```

### Step 4: Fast Reload with Cache Miss Handling

```python
def load_embeddings(model_id: str, tools):
    """Load embeddings if they exist, otherwise compute and cache them."""
    catalogue_hash = _catalogue_hash(tools)
    path = _cache_path(model_id, catalogue_hash)

    if path.is_file():
        data = np.load(path, allow_pickle=False)
        return dict(zip(data["names"], data["vectors"]))

    compute_and_save(model_id, tools)
    return load_embeddings(model_id, tools)

```

## Integrating with the Worker Pipeline

Hook the cache loader into [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) where tool schemas are prepared:

```python
from .agent.tools import build_schema
from .utils.tool_embeddings import load_embeddings


def _prepare_tool_embeddings(model_id: str, tool_fns):
    """Create fast lookup table: tool-name → embedding."""
    tool_schemas = [build_schema(fn) for fn in tool_fns]
    return load_embeddings(model_id, tool_schemas)

```

This replaces repeated embedding computation with a single dictionary lookup after the first run.

## Key Files and Their Roles

| File | Purpose | Critical Lines |
|------|---------|--------------|
| [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) | `@tool` decorator and `build_schema()`—generates JSON schemas that get embedded | Tool definitions |
| [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) | Native embedding calls via `needle_embed`; integration point for cached tool vectors | Lines 169-179 |
| [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) | In-memory caching patterns (`_decode_fn_cache`, `_rollout_cache`) | Caching reference |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Cache directory setup: `~/.cache/cactus-needle` | Lines 66-71 |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | On-disk cache operations for model files | Lines 242-258 |

## Performance Characteristics

- **First run**: Embedding computation scales linearly with catalogue size (milliseconds per tool)
- **Subsequent runs**: Loading from `.npz` via `np.load`—typically sub-100ms even for thousands of tools
- **Invalidation**: Automatic on any schema change via catalogue hash; manual cleanup by deleting `~/.cache/cactus-needle/tool_embeddings/`

## Summary

- **Hash the catalogue** to version-cache entries and prevent stale embeddings
- **Use Needle's standard cache path** (`~/.cache/cactus-needle/tool_embeddings/`) for consistency with existing on-disk storage
- **Reuse internal embedding routines** from [`_worker.py`](https://github.com/cactus-compute/needle/blob/main/_worker.py) rather than duplicating vectorization logic
- **Store as compressed NumPy archives** for fast I/O and memory mapping
- **Integrate at schema preparation time** in the worker pipeline for transparent fast reloads

## Frequently Asked Questions

### How does the cache know when tools have changed?

The `_catalogue_hash()` function computes a SHA256 hash of the canonicalized JSON representation of all tool schemas. Any modification to names, descriptions, or parameters changes the hash, forcing re-computation on next load.

### Can I use this with multiple models simultaneously?

Yes—the cache path includes `model_id` in the filename, so embeddings for different models remain isolated. Each model loads only its compatible vectors.

### What if I need to clear the cache manually?

Delete the directory `~/.cache/cactus-needle/tool_embeddings/` or specific `.npz` files. The code gracefully handles missing cache files by recomputing embeddings.

### Does this work with dynamically added tools at runtime?

If tools are added after initial load, the catalogue hash changes, triggering cache invalidation. For truly dynamic scenarios, consider a hybrid approach: load cached embeddings for the base catalogue, compute embeddings for new tools on-demand, and periodically merge back to disk.