# Handling Tool Selection with Hundreds of Tools in an Agent Context: A Scalable Architecture

> Discover how the ai-agent-book framework handles tool selection with hundreds of tools using a scalable three-layer architecture: Provider Registry, Safety Policy Gate, and Resolver Executor.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: architecture
- Published: 2026-08-18

---

**The ai-agent-book framework solves large-scale tool selection through a three-layer pipeline: a central Provider Registry for cataloging tools, a Safety Policy Gate for validation, and a Resolver/Executor for context-aware execution.**

Managing hundreds of tools in an AI agent requires more than a simple lookup table. As implemented in the bojieli/ai-agent-book repository, the solution combines **lazy loading**, **dynamic filtering**, and **strict security validation** to keep prompts lean and execution safe. This article breaks down the production-ready architecture that powers scalable tool selection.

## Core Architecture: Three Interconnected Layers

The framework organizes tool management into distinct components that work together to handle large tool catalogs without overwhelming the LLM or compromising security.

| Component | Core Responsibility | Source File |
|-----------|---------------------|-------------|
| **Provider Registry** | Maintains tool metadata, aliases, and discovery | [[`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py) |
| **Safety Policy Gate** | Validates calls against security policies before execution | [[`chapter9/harness-safety-gate/safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/harness-safety-gate/safety_policy_gate.py)](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/harness-safety-gate/safety_policy_gate.py) |
| **Resolver / Executor** | Looks up implementations and injects runtime context | [[`agentbook/providers/resolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py) |

Each layer addresses a specific scaling challenge: the registry prevents namespace collisions, the safety gate blocks dangerous executions, and the resolver enables flexible context injection.

## How Tool Selection Works Step by Step

### Step 1: Tool Discovery at Startup

When the agent initializes, every provider module calls `register()` on the global registry. This populates two critical data structures:

- **`supported_providers`** — A set of provider identifiers used by CLI tools for auto-completion
- **`tool_aliases`** — A mapping from user-friendly names to internal function objects

```python
from agentbook.providers.registry import supported_providers

print(supported_providers())

# → {'openrouter', 'local', 'anthropic', ...}

```

The registry in [`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py) handles both built-in providers (OpenRouter, local models) and custom extensions without code modification.

### Step 2: Dynamic Prompt Generation

The agent constructs the LLM prompt using `registry.get_prompt_description()`, which returns tool descriptions filtered by current context. For browser automation, this means excluding actions irrelevant to the current URL.

```python
from browser_use.tools.registry.service import Registry

registry = Registry()

# Generates a Pydantic model containing only actions applicable to the URL

ActionModel = registry.create_action_model(page_url="https://example.com")
payload = ActionModel(action="click_element_by_index", index=3)

```

As implemented in [`browser_use/tools/registry/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/tools/registry/service.py), this **dynamic action model generation** keeps the LLM's action space small even when the full catalog contains hundreds of browser operations.

### Step 3: Safety Validation Before Execution

Every tool call passes through the safety gate before reaching the resolver. The `validate_tool_call()` method in [`safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/safety_policy_gate.py) performs multiple checks:

- Disallowed command patterns (e.g., `rm -rf /`)
- Required confirmations for destructive actions
- Rate limit and token expiration validation
- Domain whitelisting for browser operations

```python
from chapter9.harness_safety_gate.safety_policy_gate import SafetyPolicyGate

gate = SafetyPolicyGate()
decision = gate.validate_tool_call(
    "delete_file",
    {"path": "important_report.docx"},
    user_confirmed=True
)
print(decision.allowed)   # True only after explicit confirmation

```

The comprehensive test suite in [[`tests/test_ch9_safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_ch9_safety_policy_gate.py)](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_ch9_safety_policy_gate.py) covers edge cases including injection attempts and privilege escalation scenarios.

### Step 4: Resolution and Execution

Validated tool calls reach `resolution.resolve_tool_call()`, which handles:

1. Looking up the concrete implementation from the registry
2. Injecting runtime dependencies (browser sessions, file handles, API credentials)
3. Executing the tool and wrapping results

```python
from agentbook.providers.resolution import resolve_tool_call

# Assume `tool_name` and `args` came from the LLM and passed safety gate

result = resolve_tool_call(tool_name="read_file", arguments=args)
print(result)   # {'content': '...'}

```

The resolver in [`agentbook/providers/resolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py) decouples the LLM's tool naming from actual implementation, enabling backward compatibility and smooth deprecation.

## Scaling Strategies for Hundreds of Tools

### Lazy Loading of Heavy Dependencies

Providers import expensive libraries—Playwright for browser automation, Docker SDK for containerized tools—only when `register()` is called. This prevents startup overhead from ballooning as the tool catalog grows.

### Context-Aware Filtering

The `get_prompt_description()` method accepts optional filters. For browser tools, passing the current `page_url` restricts available actions to those valid for that page state:

```python

# Instead of 200+ browser actions, only 15 relevant to current page

relevant_tools = registry.get_prompt_description(page_url="https://example.com/login")

```

### Alias and Namespace Management

Tools support multiple addressing schemes:

- **Short aliases**: `read_file`, `search_google`
- **Fully qualified names**: `browser_use.tools.scroll`

Aliases resolve at lookup time, allowing tool renaming without breaking existing agent configurations.

## Security Considerations at Scale

A large tool surface area increases attack vectors. The ai-agent-book framework mitigates this through:

- **Whitelist-based validation**: Only explicitly permitted patterns execute
- **Confirmation flows**: Destructive operations require user acknowledgment
- **Rate limiting**: Prevents abuse of expensive operations (LLM API calls, file writes)
- **Domain restrictions**: Browser actions validate against allowed URL patterns

These policies are enforced in [`safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/safety_policy_gate.py) before any tool executes, creating a** fail-closed** security model.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [[`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py) | Central catalog, provider discovery, alias handling |
| [[`agentbook/providers/resolution.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/resolution.py) | Tool resolution and runtime context injection |
| [[`agentbook/providers/models.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/models.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/models.py) | Data classes for model providers |
| [[`agentbook/providers/openrouter.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/openrouter.py)](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/openrouter.py) | OpenRouter API integration |
| [[`chapter9/harness-safety-gate/safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/harness-safety-gate/safety_policy_gate.py)](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/harness-safety-gate/safety_policy_gate.py) | Security validation layer |
| [[`chapter9/browser-use-rpa/browser-use/browser_use/tools/registry/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/browser-use-rpa/browser-use/browser_use/tools/registry/service.py)](https://github.com/bojieli/ai-agent-book/blob/main/chapter9/browser-use-rpa/browser-use/browser_use/tools/registry/service.py) | Browser tool registry with dynamic models |
| [[`tests/test_providers.py`](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_providers.py)](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_providers.py) | Registry invariants and extensibility tests |
| [[`tests/test_ch9_safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_ch9_safety_policy_gate.py)](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_ch9_safety_policy_gate.py) | Safety gate validation scenarios |

## Summary

- **Provider Registry** in [`agentbook/providers/registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/agentbook/providers/registry.py) creates a searchable, extensible namespace for hundreds of tools
- **Dynamic filtering** via `get_prompt_description()` keeps LLM context windows manageable by excluding irrelevant tools
- **Safety Policy Gate** enforces security policies before execution, preventing dangerous tool calls from reaching the resolver
- **Lazy loading** and **alias resolution** minimize startup overhead and enable backward-compatible tool evolution
- **Context-aware action models** in the browser registry demonstrate how domain-specific tool subsets are generated on demand

## Frequently Asked Questions

### How does the framework prevent the LLM from being overwhelmed by too many tool descriptions?

The framework uses **context-aware filtering** through `registry.get_prompt_description()`. For browser automation, passing the current `page_url` generates a Pydantic model containing only actions valid for that page state—often reducing 200+ possible actions to 15 relevant ones. This dynamic pruning happens before the prompt is constructed, keeping token usage low and decision quality high.

### What happens if an LLM generates a tool call for a dangerous operation?

The **Safety Policy Gate** intercepts all tool calls before execution. The `validate_tool_call()` method in [`safety_policy_gate.py`](https://github.com/bojieli/ai-agent-book/blob/main/safety_policy_gate.py) checks against disallowed command patterns, requires explicit user confirmation for destructive actions like `delete_file`, and enforces rate limits. The gate operates on a fail-closed principle: any call failing validation is rejected with a clear error message to the LLM.

### Can the tool catalog grow without modifying core framework code?

Yes. The **Provider Registry** uses a registration pattern where new provider modules call `register()` to add their tools. This enables third-party extensions without changing [`registry.py`](https://github.com/bojieli/ai-agent-book/blob/main/registry.py). Aliases allow tools to be renamed or reorganized while maintaining backward compatibility, and the test suite in [`tests/test_providers.py`](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_providers.py) verifies that registry invariants hold as providers are added.

### How does the browser tool registry handle pages with different available actions?

The browser registry in [`browser_use/tools/registry/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/tools/registry/service.py) implements **dynamic action model generation**. The `create_action_model()` method builds a Pydantic model on-the-fly containing only actions applicable to the current page state—clickable elements differ between a login form and a dashboard. This keeps the LLM's action space minimal while preserving full functionality across diverse page types.