How Needle Tool Retrieval Works: Optimizing for Large Tool Catalogs

Needle tool retrieval operates by attaching OpenAI-compatible JSON-Schema metadata to Python functions through the @tool decorator, enabling dynamic LLM discovery and invocation, while large catalogs require lazy schema generation, caching, and relevance filtering to maintain performance.

Needle is an open-source framework that exposes Python callables to large language models as executable tools. Understanding the internals of Needle tool retrieval is critical for developers building AI agents with extensive utility libraries, as naive implementations can introduce significant latency when handling hundreds or thousands of registered functions.

The Core Mechanism of Needle Tool Retrieval

The @tool Decorator and Schema Attachment

At the heart of Needle's architecture lies the @tool decorator defined in needle/agent/tools.py. When applied to any Python callable, this decorator immediately invokes build_schema(fn) and attaches the resulting dictionary to a special attribute _needle_tool:

def tool(fn: Callable) -> Callable:
    fn._needle_tool = build_schema(fn)   # ← attaches the schema

    return fn

This eager attachment strategy ensures that every decorated function carries its own JSON-Schema definition, which the LLM uses to understand parameter requirements and functionality.

How build_schema Constructs Tool Definitions

The build_schema function (also in needle/agent/tools.py) performs deep introspection to generate OpenAI-compatible schemas through a five-stage pipeline:

1. Signature inspection – inspect.signature(fn) extracts the complete parameter list.
2. Type hint resolution – typing.get_type_hints(fn, include_extras=True) retrieves annotations, supporting typing.Annotated and Pydantic models.
3. JSON-type mapping – The internal _json_type helper maps Python types (e.g., int, str, list) to JSON-Schema equivalents (integer, string, array).
4. Field metadata extraction – Custom Field objects allow constraints like ge (greater-than-or-equal), le, enum, and other validation rules.
5. Doc-string parsing – _parse_doc extracts human-readable descriptions and per-argument documentation.

The resulting schema follows the standard function-calling format:

{
  "name": "function_name",
  "description": "Human-readable description",
  "parameters": {
    "type": "object",
    "properties": { ... },
    "required": [ ... ]
  }
}

Runtime Tool Discovery and Execution

During an LLM inference call, Needle gathers all functions bearing the _needle_tool attribute—either by iterating over imported modules or processing a user-supplied list—and transmits their schemas to the model. When the LLM returns a tool invocation request containing a function name and arguments, Needle routes the call to the original Python callable and returns the result.

Optimizing Large Tool Catalogs in Needle

When managing hundreds or thousands of tools, eagerly building and transmitting every schema creates significant CPU and network overhead. Implement these proven strategies to maintain sub-second response times:

Implement Lazy Schema Generation

Replace the eager @tool decorator with a thin wrapper that stores only the function reference, deferring schema construction until the tool is actually referenced:

def lazy_tool(fn: Callable) -> Callable:
    """Wrap a function without building its schema immediately."""
    fn._needle_tool = None          # placeholder

    fn._needle_builder = lambda: build_schema(fn)
    return fn

def get_tool_schema(fn: Callable) -> dict:
    """Retrieve the schema, building it lazily if needed."""
    if fn._needle_tool is None:
        fn._needle_tool = fn._needle_builder()
    return fn._needle_tool

This approach allows catalogs to load instantly, with schemas materializing only when the LLM selects the specific tool.

Utilize Schema Caching

Avoid recomputing identical schemas across multiple requests by implementing a module-level cache in needle/agent/tools.py:

_schema_cache: dict[Callable, dict] = {}

def cached_build_schema(fn: Callable) -> dict:
    if fn not in _schema_cache:
        _schema_cache[fn] = build_schema(fn)
    return _schema_cache[fn]

Subsequent accesses return the cached dictionary object, eliminating introspection overhead for frequently used tools.

Filter Tools by Relevance

LLMs exhibit better accuracy and lower latency when presented with focused tool sets rather than exhaustive catalogs. Implement a select_tools helper that scores candidates against the user prompt using keyword matching or embedding similarity, retaining only the top-N most relevant items (e.g., max=20):

def select_tools(prompt: str, candidates: list[Callable], max: int = 20) -> list[dict]:
    # Scoring logic (keyword matching, embeddings, etc.)

    scored = [(score_tool(c, prompt), c) for c in candidates]
    scored.sort(reverse=True)
    return [get_tool_schema(fn) for _, fn in scored[:max]]

Advanced Performance Techniques

Parallel Schema Building: Utilize concurrent.futures.ThreadPoolExecutor to compute schemas for batches of functions concurrently during initial catalog warm-up, then merge the results.

Compress Schema Payloads: Strip optional fields (such as empty description entries) from the JSON structure, and optionally apply gzip compression before transmitting to the LLM service to reduce network latency.

Versioned Catalog Caching: Attach a SHA-256 hash of each schema (using hashlib.sha256(json.dumps(schema).encode()).hexdigest()) and track versions across sessions. Only resend schemas whose hashes differ from the previous request, minimizing redundant data transmission.

Batch Loading for Reduced Import Overhead

Group tool modules into logical packages and import them lazily using importlib.import_module only when a tool from that specific package is selected by the relevance filter. This prevents loading hundreds of unused modules at startup.

Summary

• Needle tool retrieval depends on the @tool decorator in needle/agent/tools.py, which attaches JSON-Schema metadata to the _needle_tool attribute of Python functions.
• The build_schema function uses inspect.signature, typing.get_type_hints, and _parse_doc to generate OpenAI-compatible definitions supporting Pydantic models and custom Field constraints.
• For large catalogs, replace eager schema generation with lazy loading using placeholder attributes and on-demand builders.
• Implement a _SCHEMA_CACHE dictionary to avoid recomputing schemas across multiple LLM calls.
• Reduce payload size through relevance filtering, compression, and versioned hashing to optimize network performance and LLM context window utilization.

Frequently Asked Questions

What is the Needle tool retrieval process?

Needle tool retrieval works by scanning for functions marked with the _needle_tool attribute (set by the @tool decorator), collecting their JSON-Schema definitions, and passing them to the LLM during inference. When the model returns a tool call, Needle matches the function name to the original callable and executes it with the provided arguments.

How does the @tool decorator generate JSON schemas?

The decorator invokes build_schema(fn), which introspects the function using inspect.signature(fn) and typing.get_type_hints(fn, include_extras=True), maps Python types to JSON-Schema types via _json_type, and extracts documentation through _parse_doc. The resulting dictionary conforms to the OpenAI function-calling specification with name, description, and parameters keys.

What are the best practices for optimizing large tool catalogs in Needle?

For catalogs containing hundreds or thousands of tools, implement lazy schema generation to defer build_schema calls until tools are actually needed, use a module-level _SCHEMA_CACHE to store computed schemas, and apply relevance filtering to send only the top-N most applicable tools to the LLM. Additionally, use parallel processing for initial catalog loading and compress schemas to reduce network overhead.

Where is the schema building logic implemented in Needle?

The core implementation resides in needle/agent/tools.py, which contains the tool decorator, build_schema function, Field class for metadata constraints, and helper functions _json_type and _parse_doc. Unit tests demonstrating decorator behavior are available in tests/test_tools.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}