How to Implement Scalable Tool Discovery for AI Agents Managing Hundreds of Tools

Implement scalable tool discovery for AI agents by combining a lightweight JSON-based ToolLibrary with keyword search, lazy loading, and sandboxed execution to safely manage hundreds of tools without pre-loading everything into memory.

The bojieli/ai-agent-book repository demonstrates a production-ready approach to tool discovery for AI agents that scales effortlessly to hundreds of registered capabilities. Instead of loading all tools into the context window or memory, the system uses a self-evolving file-based architecture that lets agents discover, create, validate, and reuse tools dynamically. This approach eliminates context limitations while maintaining fast lookup and safe execution.

Core Architecture of the Tool Discovery System

The ToolLibrary Metadata Store

At the heart of the system is the ToolLibrary class defined in [tool_manager.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolving-tools/tool_manager.py#L43-L53) (lines 43-53). Each tool is stored as a lightweight JSON file containing its metadata and executable Python code. This file-system approach avoids database dependencies and keeps the architecture portable across environments.

Tool Creation and Schema Normalization

The create_tool method (lines 50-99) wraps user-provided functions as OpenAI-compatible tools. It enforces quality through a two-stage validation process: first, a syntax compile check, then an optional sandbox execution via _run_record. The method also calls normalize_schema to ensure every tool conforms to OpenAI's expected JSON schema, preventing costly runtime 400 errors when scaling to many tools.

Keyword-Based Tool Search

For discovery, the search_tools method (lines 105-124) performs simple keyword tokenization over tool names and descriptions. While this implements an O(N) linear scan, the constant factor remains extremely low because JSON files are tiny and disk I/O stays cheap even when managing hundreds of tools.

Sandboxed Execution

The execute_tool method (lines 142-152) runs stored run(**kwargs) functions in isolated subprocesses using _run_python_source. This sandbox approach injects a custom PYTHONPATH and temporary execution environment, ensuring that misbehaving tools cannot crash the agent process—a critical safety feature when executing hundreds of tools concurrently.

Scaling Mechanisms for Hundreds of Tools

The repository implements several specific strategies to handle scale efficiently:

• Metadata Indexing: Uses simple keyword tokenization (query → terms) with linear scan over JSON files. This avoids the complexity of vector databases while maintaining acceptable performance for hundreds of small metadata files.

• Lazy Loading: Tools are read from disk only when needed via get_tool, not pre-loaded into memory. This keeps the memory footprint constant regardless of how many tools are registered in the library.

• Pre-validation Gate: The syntax compile and sandbox run before persisting ensures bad tools never pollute the library. This saves storage space and prevents cascading failures when many tools are available.

• Unlocking Mechanism: After search_tools finds a tool, its name is added to the _unlocked set, allowing immediate subsequent calls. This avoids unnecessary re-searches during multi-step agent workflows.

The Self-Evolving Agent Loop

The SelfEvolvingAgent class in [agent.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolving-tools/agent.py#L92-L101) (lines 92-101) orchestrates the complete discovery workflow:

1. Search: The agent first calls search_tools to check for existing capabilities.
2. Generate: When no matching tool exists, it uses web_search, code_interpreter, and LLM generation to create new code.
3. Validate: The new tool undergoes validation via create_tool before registration.
4. Execute: Once registered, subsequent tasks reuse the tool directly via execute_tool without recreation.

The demonstration in [demo.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/self-evolving-tools/demo.py#L57-L72) (lines 57-72) showcases this evolution-reuse cycle: the first task creates the tool, while the second task discovers and reuses it immediately.

Implementation Examples

Create a new tool with automatic validation:

from tool_manager import ToolLibrary
lib = ToolLibrary()

code = """
def run(start, end):
    from datetime import date
    s = date.fromisoformat(start)
    e = date.fromisoformat(end)
    return {"days": (e - s).days}
"""

params = {
    "type": "object",
    "properties": {
        "start": {"type": "string", "description": "ISO start date"},
        "end":   {"type": "string", "description": "ISO end date"},
    },
    "required": ["start", "end"],
}

# Optional test_args trigger the pre-validation sandbox run

result = lib.create_tool(
    name="days_between",
    description="Calculate days between two ISO dates",
    parameters=params,
    code=code,
    test_args={"start": "2020-01-01", "end": "2020-01-10"},
)
print(result)  # → {"success": True, ...}

Search for existing tools by keyword:

search = lib.search_tools("date days between")
print(search["tools"])  # → [{"name": "days_between", ...}]

Execute the registered tool in isolation:

out = lib.execute_tool("days_between", {"start": "2021-03-01", "end": "2021-03-15"})
print(out["result"])  # → {"days": 14}

Agent-level automatic discovery and creation:

from agent import SelfEvolvingAgent
agent = SelfEvolvingAgent(allow_create=True, model="gpt-4")
answer = agent.run("How many days between 2022-05-01 and 2022-05-20?")

# Agent discovers, creates, and reuses the tool automatically

print(answer)

Summary

• File-based storage enables O(N) keyword search with minimal overhead, avoiding database complexity while scaling to hundreds of tools.
• Pre-validation gates using syntax checks and sandbox execution prevent invalid tools from entering the library, ensuring system stability.
• Lazy loading maintains constant memory footprint by reading tools from disk only when requested via get_tool.
• Subprocess sandboxing isolates execution environments to prevent tool failures from crashing the agent process.
• Unlock tracking eliminates redundant searches by caching discovered tool availability in the _unlocked set.

Frequently Asked Questions

How does keyword search scale to hundreds of tools without vector databases?

The search_tools implementation uses simple keyword tokenization and linear scanning over JSON files. While technically O(N), the approach remains performant because each tool metadata file is only a few kilobytes, making disk I/O negligible for hundreds of files. The repository prioritizes simplicity and low constant factors over the added complexity of vector search infrastructure for this scale.

What prevents malicious or buggy tools from crashing the agent?

The system implements defense in depth through the execute_tool method's subprocess isolation. Each tool runs in a separate Python process with controlled PYTHONPATH via _run_python_source, ensuring memory leaks or exceptions in one tool cannot affect the agent's main process. Additionally, the create_tool validation gate catches syntax errors and runtime exceptions during the sandbox test phase before the tool ever enters the library.

Can the ToolLibrary handle concurrent tool creation and execution?

Yes, because each tool is stored as an independent JSON file on the filesystem. The ToolLibrary does not maintain an in-memory index that requires locking, and tool execution occurs in separate subprocesses. However, concurrent writes to the same tool file should be managed at the application level, as the base implementation focuses on single-agent usage patterns.

How does the unlocking mechanism improve performance?

After the agent successfully calls search_tools and finds a matching tool, the tool name is added to the _unlocked set within the SelfEvolvingAgent class. On subsequent operations, the agent checks this set first before searching, effectively creating a hot-cache of available capabilities. This prevents repeated linear scans when the same tool is needed multiple times during a conversation, reducing the search overhead to O(1) for frequently used tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}