How Tool Schemas Are Managed in Needle 2's KV Pools

Needle 2 stores agent tool definitions directly in the same key-value (KV) cache used for model activations, enabling sub-millisecond retrieval through grouped memory access and unified quantisation pipelines.

The cactus-compute/needle repository eliminates external database lookups for tool definitions by co-locating schemas with transformer state. This architecture treats tool schemas as first-class citizens in Needle 2's KV pools, allowing the runtime to fetch tool definitions using the same high-throughput matrix operations that drive the attention mechanism.

The Architecture of Tool Schema Storage

Unified Memory Layout with Model Activations

Unlike traditional agent frameworks that query external registries, Needle 2 serializes tool schemas into fixed-size tensors and writes them directly into the KV cache. In needle/model/architecture.py, the system defines KV_BUDGET_BYTES (approximately 11 MiB) to cap total cache memory across both activations and tool entries. The KV_GROUP constant (defaulting to 32) organizes these entries into contiguous blocks, enabling single-memory-read fetching of entire schema batches. The kv_window() function dynamically calculates the maximum sequence length available given the current tool schema memory consumption, automatically shrinking the context window when the cache population increases.

Quantisation and Memory Efficiency

Tool schemas undergo identical compression to model key-value pairs. In needle/model/quantize.py, the cq_fake_quant_kv() function applies configurable bit-width reduction controlled by KV_BITS (default 0 for no quantisation). When enabled, this shared quantisation routine reduces schema storage overhead while preserving the de-quantisation pathways used during the forward pass, ensuring zero-latency overhead when switching between tool retrieval and attention computation.

Loading and Serializing Tool Schemas

Discovery via load_tool_schemas()

The primary entry point for schema ingestion resides in needle/agent/tools.py. The load_tool_schemas(path) function traverses directories containing JSON or YAML schema definitions, converting each tool's name, description, and parameter specifications into compact byte representations. This discovery mechanism executes at startup, walking the provided directory and preparing tensors for KV insertion.

KV Write Operations and Grouping

After serialization, schemas pass through the KVWriter class, which manages slot allocation within the cache. The writer maintains an internal pointer to track free space, advancing as it writes tensors via write(tensor, slot) calls. Schemas are packed into groups of KV_GROUP slots (32 entries per group), ensuring that related tool definitions occupy contiguous memory regions. This layout allows the runtime to fetch an entire toolkit with a single indexing operation: kv = cache[:, group_idx, ...].

Runtime Retrieval and Dynamic Updates

Group-Based Lookup During Inference

During the forward pass, the model extracts tool schemas using standard KV cache indexing. When a tool-call token is generated, the runtime maps the tool identifier to a specific KV group index, retrieves the raw bytes, and applies de-quantisation using the inverse of cq_fake_quant_kv(). The resulting tensor feeds directly into the tool dispatch logic without CPU-GPU transfer overhead, maintaining the throughput characteristics of native attention operations.

Hot-Swapping Schemas Without Model Restart

Because schemas reside in standard KV slots, needle/agent/tools.py supports dynamic updates by overwriting specific cache entries. New plugins register by calling next_free_slot() to obtain an unused index, or by reusing existing indices to update definitions. This capability enables runtime tool modification without model reinitialization, KV cache eviction, or interruption to active inference sessions.

Implementation Examples

Register a directory of tool schemas at startup:

from needle.agent.tools import load_tool_schemas

# Scans ./my_tools for JSON/YAML schema files and loads into KV cache

load_tool_schemas("./my_tools")

Manually insert a single schema for dynamic plugin registration:

import torch
from needle.agent.tools import KVWriter
from needle.model.quantize import cq_fake_quant_kv

schema = {
    "name": "weather",
    "description": "Get current weather for a city",
    "parameters": {"city": {"type": "string", "description": "City name"}}
}

# Serialize to fixed-size tensor (default torch.float16)

tensor = torch.tensor([hash(str(schema)) % (2**16)], dtype=torch.float16).unsqueeze(0)

# Optional quantisation if KV_BITS > 0

if KV_BITS:
    tensor = cq_fake_quant_kv(tensor, KV_BITS, KV_GROUP)

# Write to next available KV slot

writer = KVWriter()
writer.write(tensor, slot=writer.next_free_slot())

Retrieve and de-quantise a schema during inference:

def get_schema_for_tool(tool_name: str, kv_cache) -> dict:
    # Map tool name to KV group index (internal lookup)

    group_idx = kv_cache.lookup_group(tool_name)
    
    # Fetch raw KV bytes for the entire group

    kv_blob = kv_cache[:, group_idx, ...]
    
    # De-quantise using same routine as model KV entries

    schema_tensor = dequantize(kv_blob, KV_BITS)
    
    return deserialize_schema(schema_tensor)

Summary

• Tool schemas in Needle 2's KV pools share memory with model activations, eliminating external database latency.
• The load_tool_schemas() function in needle/agent/tools.py handles bulk ingestion of JSON/YAML definitions at startup.
• KV_BUDGET_BYTES and kv_window() enforce memory limits while balancing sequence length against tool count.
• cq_fake_quant_kv() applies optional compression to schemas using the same pipeline as model KV entries.
• Dynamic updates occur via KVWriter operations that overwrite cache slots without requiring model restart.

Frequently Asked Questions

What file handles tool schema discovery in Needle 2?

The needle/agent/tools.py module contains the load_tool_schemas() function, which scans directories for JSON or YAML schema files and serializes them into the KV cache.

How does Needle 2 prevent tool schemas from exhausting available memory?

The system enforces KV_BUDGET_BYTES (approximately 11 MiB) across all cache entries. The kv_window() function in needle/model/architecture.py dynamically recalculates the maximum supported sequence length based on current tool schema consumption, ensuring model activations retain sufficient space.

Can tool schemas be updated dynamically without restarting the model?

Yes. Because schemas occupy standard KV slots, the KVWriter class can overwrite existing entries or utilize next_free_slot() to register new tools at runtime. This enables hot-swapping of tool definitions without cache invalidation or model reinitialization.

What quantisation method applies to tool schemas in the KV pool?

Needle 2 uses cq_fake_quant_kv() from needle/model/quantize.py, applying the same bit-width reduction controlled by KV_BITS that is used for model activations. This ensures uniform memory layout and shared de-quantisation pathways during inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}