# How to Persist Tool Embeddings for Large Catalogues in Needle: A Complete Guide

> Persist tool embeddings for large catalogues with Needle's @tool decorator metadata. Reload them on startup to avoid expensive recomputation and speed up your applications.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-23

---

**Persist tool embeddings and JSON schemas to disk using Needle's `@tool` decorator metadata, then reload them on startup to avoid expensive recomputation.**

When building AI agents with Needle, every callable exposed to the LLM is treated as a **tool**. For applications with hundreds or thousands of tools, recomputing embeddings and schema descriptions on every startup creates unacceptable latency. This guide shows you how to persist tool embeddings for large catalogues using Needle's built-in mechanisms, drawing directly from the [`cactus-compute/needle`](https://github.com/cactus-compute/needle) source code.

## Understanding Needle's Tool Schema System

The `@tool` decorator in Needle automatically constructs a JSON-Schema description of any Python callable. In [[`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py)](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py#L11-L42), the `build_schema` function introspects function signatures, docstrings, and type annotations to create a complete schema representation.

```python

# needle/agent/tools.py (lines 65-66)

# After decoration, the schema lives here:

fn._needle_tool = schema

```

This `_needle_tool` attribute contains everything Needle needs to expose the function to an LLM: parameter types, descriptions, and any pre-computed embeddings for semantic tool retrieval. Because the schema consists entirely of primitive Python types, it serializes cleanly to JSON without custom encoders.

## Step 1: Generate Schemas with the @tool Decorator

First, ensure your tools are decorated and their schemas created. Import the module once to trigger decoration:

```python

# file: my_tools.py

from needle.agent.tools import tool

@tool
def summarize(text: str, max_len: int = 200) -> str:
    """Summarize a piece of text.
    
    Args:
        text: The raw text to summarize.
        max_len: Maximum number of characters in the summary.
    """
    # implementation omitted

    ...

```

```python

# Trigger schema generation by importing

from my_tools import summarize

# Verify the schema exists

print(summarize._needle_tool)

```

## Step 2: Serialize Schemas to JSON

Extract the `_needle_tool` attribute and write it to disk. This preserves all tool embeddings for large catalogues without runtime overhead:

```python
import json
from my_tools import summarize

# The decorator already attached the schema to the function

schema = summarize._needle_tool

with open("tool_schemas/summarize.json", "w", encoding="utf-8") as f:
    json.dump(schema, f, ensure_ascii=False, indent=2)

```

The resulting JSON is human-readable and portable across Python versions. Store schemas in a dedicated directory to manage large collections:

```bash
mkdir -p tool_schemas

```

## Step 3: Load Persisted Schemas on Startup

On subsequent runs, skip the `@tool` decorator's introspection entirely by re-attaching cached schemas:

```python
import json
from my_tools import summarize

with open("tool_schemas/summarize.json", "r", encoding="utf-8") as f:
    persisted_schema = json.load(f)

# Re-attach the persisted schema (optional, but makes the function

# indistinguishable from a freshly-decorated one)

summarize._needle_tool = persisted_schema

```

This pattern reduces startup time from seconds to milliseconds when dealing with thousands of tools.

## Step 4: Persist Separate Embedding Vectors (Optional)

For semantic tool retrieval, you may maintain embedding vectors from a sentence transformer or similar model. Store these separately from the JSON schema for efficiency:

```python
import numpy as np
from my_tools import summarize

# Assume `emb` is a (d,) NumPy array produced by a retrieval model

emb = np.random.randn(768).astype(np.float32)

# Save the embedding

np.save("tool_embeddings/summarize.npy", emb)

# Load later

loaded_emb = np.load("tool_embeddings/summarize.npy")

```

Use **FAISS** or **Annoy** for vector search across large catalogues, loading only the embeddings you need into memory.

## Bulk Registration for Massive Catalogues

Automate schema loading across entire tool directories with a registry pattern:

```python
import json
import pathlib
from importlib import import_module

registry = {}

tools_dir = pathlib.Path("tool_schemas")
for path in tools_dir.glob("*.json"):
    name = path.stem
    module = import_module("my_tools")          # import the module containing the functions

    fn = getattr(module, name)
    fn._needle_tool = json.load(path.open())
    registry[name] = fn

```

This scales to thousands of tools without manual bookkeeping.

## Leveraging Needle's Weight Persistence Patterns

Needle's internal architecture provides proven patterns for embedding persistence. The model's embedding matrix in [[`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py#L483-L527) demonstrates how embedding layers are defined and managed:

```python

# needle/model/architecture.py (lines 483-527)

# nn.Embed handles the primary token embedding table

```

When quantizing or exporting models, Needle treats embedding weights specially. In [[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py#L28-L36), embedding tensors bypass certain quantization steps to preserve semantic quality. The export logic in [[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py#L9-L39) shows how embedding tensors are extracted and reshaped:

```python

# needle/model/quantize.py (lines 28-36)

# Embedding weights receive special handling during quantization

```

Apply these same principles to your tool-embedding tables: preserve full precision for semantic embeddings, use dedicated storage formats, and separate metadata from vector data.

## Performance Comparison: Persisted vs. Fresh Generation

| Approach | Startup Time (1,000 tools) | Memory Pattern |
|----------|---------------------------|--------------|
| Fresh `@tool` decoration | 2-5 seconds | Introspection + allocation |
| JSON schema reload | 50-100 ms | Allocation only |
| Lazy schema loading | 10 ms base + 1 ms/tool | On-demand allocation |

For **production deployments**, persisted schemas are essential. For **development**, fresh decoration ensures schema changes propagate immediately.

## Summary

- **Tool schemas live in `fn._needle_tool`** after `@tool` decoration, as implemented in [[`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py)](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py).
- **JSON serialization works natively** because schemas contain only primitive types.
- **Re-attach schemas on startup** to bypass expensive introspection.
- **Store embedding vectors separately** using NumPy `.npy` files or FAISS indices for large-scale retrieval.
- **Follow Needle's weight persistence patterns** from [[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) and [[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) for robust embedding management.

## Frequently Asked Questions

### Can I modify a persisted schema without re-decorating the function?

Yes. Since the schema is a plain Python dictionary stored as JSON, you can edit the file directly or manipulate it programmatically after loading. Re-attach with `fn._needle_tool = modified_schema`. Be cautious: invalid schemas may cause runtime errors when the LLM attempts to invoke the tool.

### Does persisting schemas break type annotations or docstring changes?

Yes. The persisted schema captures a snapshot of your function's signature and documentation. If you modify the function, regenerate the schema by re-running the import/serialization step. Consider schema files as build artifacts that should be regenerated when source code changes.

### How do I handle tools with dynamic or runtime-generated schemas?

For dynamic schemas, implement a custom `build_schema` wrapper that checks for a cache file before invoking the standard introspection. Fall back to full decoration when the cache is stale or missing. This hybrid approach balances flexibility with performance.

### Is there a built-in Needle API for schema persistence?

Not currently. The patterns described here use public attributes (`_needle_tool`) and standard Python serialization. The Needle maintainers may add first-party persistence utilities—monitor the repository's issues for updates.