# How Tool Schemas Are Managed in Needle 2's KV Pools

> Learn how Needle 2 manages tool schemas in KV pools for fast agent tool retrieval. Discover sub-millisecond access and unified quantisation.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-17

---

**Needle 2 stores agent tool definitions directly in the same key-value (KV) cache used for model activations, enabling sub-millisecond retrieval through grouped memory access and unified quantisation pipelines.**

The cactus-compute/needle repository eliminates external database lookups for tool definitions by co-locating schemas with transformer state. This architecture treats tool schemas as first-class citizens in Needle 2's KV pools, allowing the runtime to fetch tool definitions using the same high-throughput matrix operations that drive the attention mechanism.

## The Architecture of Tool Schema Storage

### Unified Memory Layout with Model Activations

Unlike traditional agent frameworks that query external registries, Needle 2 serializes tool schemas into fixed-size tensors and writes them directly into the KV cache. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the system defines **`KV_BUDGET_BYTES`** (approximately 11 MiB) to cap total cache memory across both activations and tool entries. The **`KV_GROUP`** constant (defaulting to 32) organizes these entries into contiguous blocks, enabling single-memory-read fetching of entire schema batches. The `kv_window()` function dynamically calculates the maximum sequence length available given the current tool schema memory consumption, automatically shrinking the context window when the cache population increases.

### Quantisation and Memory Efficiency

Tool schemas undergo identical compression to model key-value pairs. In [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), the **`cq_fake_quant_kv()`** function applies configurable bit-width reduction controlled by **`KV_BITS`** (default 0 for no quantisation). When enabled, this shared quantisation routine reduces schema storage overhead while preserving the de-quantisation pathways used during the forward pass, ensuring zero-latency overhead when switching between tool retrieval and attention computation.

## Loading and Serializing Tool Schemas

### Discovery via `load_tool_schemas()`

The primary entry point for schema ingestion resides in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py). The **`load_tool_schemas(path)`** function traverses directories containing JSON or YAML schema definitions, converting each tool's name, description, and parameter specifications into compact byte representations. This discovery mechanism executes at startup, walking the provided directory and preparing tensors for KV insertion.

### KV Write Operations and Grouping

After serialization, schemas pass through the **`KVWriter`** class, which manages slot allocation within the cache. The writer maintains an internal pointer to track free space, advancing as it writes tensors via **`write(tensor, slot)`** calls. Schemas are packed into groups of `KV_GROUP` slots (32 entries per group), ensuring that related tool definitions occupy contiguous memory regions. This layout allows the runtime to fetch an entire toolkit with a single indexing operation: `kv = cache[:, group_idx, ...]`.

## Runtime Retrieval and Dynamic Updates

### Group-Based Lookup During Inference

During the forward pass, the model extracts tool schemas using standard KV cache indexing. When a tool-call token is generated, the runtime maps the tool identifier to a specific KV group index, retrieves the raw bytes, and applies de-quantisation using the inverse of `cq_fake_quant_kv()`. The resulting tensor feeds directly into the tool dispatch logic without CPU-GPU transfer overhead, maintaining the throughput characteristics of native attention operations.

### Hot-Swapping Schemas Without Model Restart

Because schemas reside in standard KV slots, [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) supports dynamic updates by overwriting specific cache entries. New plugins register by calling **`next_free_slot()`** to obtain an unused index, or by reusing existing indices to update definitions. This capability enables runtime tool modification without model reinitialization, KV cache eviction, or interruption to active inference sessions.

## Implementation Examples

Register a directory of tool schemas at startup:

```python
from needle.agent.tools import load_tool_schemas

# Scans ./my_tools for JSON/YAML schema files and loads into KV cache

load_tool_schemas("./my_tools")

```

Manually insert a single schema for dynamic plugin registration:

```python
import torch
from needle.agent.tools import KVWriter
from needle.model.quantize import cq_fake_quant_kv

schema = {
    "name": "weather",
    "description": "Get current weather for a city",
    "parameters": {"city": {"type": "string", "description": "City name"}}
}

# Serialize to fixed-size tensor (default torch.float16)

tensor = torch.tensor([hash(str(schema)) % (2**16)], dtype=torch.float16).unsqueeze(0)

# Optional quantisation if KV_BITS > 0

if KV_BITS:
    tensor = cq_fake_quant_kv(tensor, KV_BITS, KV_GROUP)

# Write to next available KV slot

writer = KVWriter()
writer.write(tensor, slot=writer.next_free_slot())

```

Retrieve and de-quantise a schema during inference:

```python
def get_schema_for_tool(tool_name: str, kv_cache) -> dict:
    # Map tool name to KV group index (internal lookup)

    group_idx = kv_cache.lookup_group(tool_name)
    
    # Fetch raw KV bytes for the entire group

    kv_blob = kv_cache[:, group_idx, ...]
    
    # De-quantise using same routine as model KV entries

    schema_tensor = dequantize(kv_blob, KV_BITS)
    
    return deserialize_schema(schema_tensor)

```

## Summary

- **Tool schemas in Needle 2's KV pools** share memory with model activations, eliminating external database latency.
- The **`load_tool_schemas()`** function in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) handles bulk ingestion of JSON/YAML definitions at startup.
- **`KV_BUDGET_BYTES`** and **`kv_window()`** enforce memory limits while balancing sequence length against tool count.
- **`cq_fake_quant_kv()`** applies optional compression to schemas using the same pipeline as model KV entries.
- Dynamic updates occur via **`KVWriter`** operations that overwrite cache slots without requiring model restart.

## Frequently Asked Questions

### What file handles tool schema discovery in Needle 2?

The [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) module contains the **`load_tool_schemas()`** function, which scans directories for JSON or YAML schema files and serializes them into the KV cache.

### How does Needle 2 prevent tool schemas from exhausting available memory?

The system enforces **`KV_BUDGET_BYTES`** (approximately 11 MiB) across all cache entries. The `kv_window()` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) dynamically recalculates the maximum supported sequence length based on current tool schema consumption, ensuring model activations retain sufficient space.

### Can tool schemas be updated dynamically without restarting the model?

Yes. Because schemas occupy standard KV slots, the **`KVWriter`** class can overwrite existing entries or utilize **`next_free_slot()`** to register new tools at runtime. This enables hot-swapping of tool definitions without cache invalidation or model reinitialization.

### What quantisation method applies to tool schemas in the KV pool?

Needle 2 uses **`cq_fake_quant_kv()`** from [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), applying the same bit-width reduction controlled by **`KV_BITS`** that is used for model activations. This ensures uniform memory layout and shared de-quantisation pathways during inference.