Handling Tool Selection with Hundreds of Tools in an Agent Context: A Scalable Architecture
The ai-agent-book framework solves large-scale tool selection through a three-layer pipeline: a central Provider Registry for cataloging tools, a Safety Policy Gate for validation, and a Resolver/Executor for context-aware execution.
Managing hundreds of tools in an AI agent requires more than a simple lookup table. As implemented in the bojieli/ai-agent-book repository, the solution combines lazy loading, dynamic filtering, and strict security validation to keep prompts lean and execution safe. This article breaks down the production-ready architecture that powers scalable tool selection.
Core Architecture: Three Interconnected Layers
The framework organizes tool management into distinct components that work together to handle large tool catalogs without overwhelming the LLM or compromising security.
| Component | Core Responsibility | Source File |
|---|---|---|
| Provider Registry | Maintains tool metadata, aliases, and discovery | [agentbook/providers/registry.py](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py) |
| Safety Policy Gate | Validates calls against security policies before execution | [chapter9/harness-safety-gate/safety_policy_gate.py](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/harness-safety-gate/safety_policy_gate.py) |
| Resolver / Executor | Looks up implementations and injects runtime context | [agentbook/providers/resolution.py](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py) |
Each layer addresses a specific scaling challenge: the registry prevents namespace collisions, the safety gate blocks dangerous executions, and the resolver enables flexible context injection.
How Tool Selection Works Step by Step
Step 1: Tool Discovery at Startup
When the agent initializes, every provider module calls register() on the global registry. This populates two critical data structures:
supported_providers— A set of provider identifiers used by CLI tools for auto-completiontool_aliases— A mapping from user-friendly names to internal function objects
from agentbook.providers.registry import supported_providers
print(supported_providers())
# → {'openrouter', 'local', 'anthropic', ...}
The registry in agentbook/providers/registry.py handles both built-in providers (OpenRouter, local models) and custom extensions without code modification.
Step 2: Dynamic Prompt Generation
The agent constructs the LLM prompt using registry.get_prompt_description(), which returns tool descriptions filtered by current context. For browser automation, this means excluding actions irrelevant to the current URL.
from browser_use.tools.registry.service import Registry
registry = Registry()
# Generates a Pydantic model containing only actions applicable to the URL
ActionModel = registry.create_action_model(page_url="https://example.com")
payload = ActionModel(action="click_element_by_index", index=3)
As implemented in browser_use/tools/registry/service.py, this dynamic action model generation keeps the LLM's action space small even when the full catalog contains hundreds of browser operations.
Step 3: Safety Validation Before Execution
Every tool call passes through the safety gate before reaching the resolver. The validate_tool_call() method in safety_policy_gate.py performs multiple checks:
- Disallowed command patterns (e.g.,
rm -rf /) - Required confirmations for destructive actions
- Rate limit and token expiration validation
- Domain whitelisting for browser operations
from chapter9.harness_safety_gate.safety_policy_gate import SafetyPolicyGate
gate = SafetyPolicyGate()
decision = gate.validate_tool_call(
"delete_file",
{"path": "important_report.docx"},
user_confirmed=True
)
print(decision.allowed) # True only after explicit confirmation
The comprehensive test suite in [tests/test_ch9_safety_policy_gate.py](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_ch9_safety_policy_gate.py) covers edge cases including injection attempts and privilege escalation scenarios.
Step 4: Resolution and Execution
Validated tool calls reach resolution.resolve_tool_call(), which handles:
- Looking up the concrete implementation from the registry
- Injecting runtime dependencies (browser sessions, file handles, API credentials)
- Executing the tool and wrapping results
from agentbook.providers.resolution import resolve_tool_call
# Assume `tool_name` and `args` came from the LLM and passed safety gate
result = resolve_tool_call(tool_name="read_file", arguments=args)
print(result) # {'content': '...'}
The resolver in agentbook/providers/resolution.py decouples the LLM's tool naming from actual implementation, enabling backward compatibility and smooth deprecation.
Scaling Strategies for Hundreds of Tools
Lazy Loading of Heavy Dependencies
Providers import expensive libraries—Playwright for browser automation, Docker SDK for containerized tools—only when register() is called. This prevents startup overhead from ballooning as the tool catalog grows.
Context-Aware Filtering
The get_prompt_description() method accepts optional filters. For browser tools, passing the current page_url restricts available actions to those valid for that page state:
# Instead of 200+ browser actions, only 15 relevant to current page
relevant_tools = registry.get_prompt_description(page_url="https://example.com/login")
Alias and Namespace Management
Tools support multiple addressing schemes:
- Short aliases:
read_file,search_google - Fully qualified names:
browser_use.tools.scroll
Aliases resolve at lookup time, allowing tool renaming without breaking existing agent configurations.
Security Considerations at Scale
A large tool surface area increases attack vectors. The ai-agent-book framework mitigates this through:
- Whitelist-based validation: Only explicitly permitted patterns execute
- Confirmation flows: Destructive operations require user acknowledgment
- Rate limiting: Prevents abuse of expensive operations (LLM API calls, file writes)
- Domain restrictions: Browser actions validate against allowed URL patterns
These policies are enforced in safety_policy_gate.py before any tool executes, creating a** fail-closed** security model.
Key Implementation Files
Summary
- Provider Registry in
agentbook/providers/registry.pycreates a searchable, extensible namespace for hundreds of tools - Dynamic filtering via
get_prompt_description()keeps LLM context windows manageable by excluding irrelevant tools - Safety Policy Gate enforces security policies before execution, preventing dangerous tool calls from reaching the resolver
- Lazy loading and alias resolution minimize startup overhead and enable backward-compatible tool evolution
- Context-aware action models in the browser registry demonstrate how domain-specific tool subsets are generated on demand
Frequently Asked Questions
How does the framework prevent the LLM from being overwhelmed by too many tool descriptions?
The framework uses context-aware filtering through registry.get_prompt_description(). For browser automation, passing the current page_url generates a Pydantic model containing only actions valid for that page state—often reducing 200+ possible actions to 15 relevant ones. This dynamic pruning happens before the prompt is constructed, keeping token usage low and decision quality high.
What happens if an LLM generates a tool call for a dangerous operation?
The Safety Policy Gate intercepts all tool calls before execution. The validate_tool_call() method in safety_policy_gate.py checks against disallowed command patterns, requires explicit user confirmation for destructive actions like delete_file, and enforces rate limits. The gate operates on a fail-closed principle: any call failing validation is rejected with a clear error message to the LLM.
Can the tool catalog grow without modifying core framework code?
Yes. The Provider Registry uses a registration pattern where new provider modules call register() to add their tools. This enables third-party extensions without changing registry.py. Aliases allow tools to be renamed or reorganized while maintaining backward compatibility, and the test suite in tests/test_providers.py verifies that registry invariants hold as providers are added.
How does the browser tool registry handle pages with different available actions?
The browser registry in browser_use/tools/registry/service.py implements dynamic action model generation. The create_action_model() method builds a Pydantic model on-the-fly containing only actions applicable to the current page state—clickable elements differ between a login form and a dashboard. This keeps the LLM's action space minimal while preserving full functionality across diverse page types.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" 
Maintain an open-source project? Get it listed too →