# How to Implement Tool Discovery and Planning Engines for Complex Agent Systems

> Learn to implement tool discovery and planning engines for complex agent systems. Build a modular pipeline for dynamic tool scanning, context-aware planning, and safe execution.

- Repository: [davidkimai/context-engineering](https://github.com/davidkimai/context-engineering)
- Tags: tutorial
- Published: 2026-02-28

---

**Implement tool discovery and planning engines for complex agent systems by creating a modular pipeline where a discovery service dynamically scans for tool specifications, a planning engine assembles context-aware execution sequences using strategies like "guided_discovery," and a sandboxed execution layer enforces safety policies.**

The `davidkimai/context-engineering` repository provides a reference implementation for building autonomous agents that can introspect their environment and self-configure at runtime. This guide explains how to implement tool discovery and planning engines for complex agent systems using the modular architecture found in the codebase, covering everything from dynamic module scanning to strategy-driven plan generation.

## Understanding the Architecture of Tool Discovery and Planning Engines

### High-Level Data Flow

The system processes user requests through a five-stage pipeline that separates discovery from execution:

```

User request ──► Problem Analyzer
               │
               ▼
   Tool Discovery Service  ◄──► Tool Registry
               │
               ▼
     Planning Engine  ◄──► Strategy Selector (e.g., "guided_discovery")
               │
               ▼
   Execution Engine (sandboxed)  ◄──► Permissions & Safety Layer
               │
               ▼
   Result Synthesizer ──► Final response

```

The **Problem Analyzer** extracts intents and domain constraints from natural language inputs. The **Tool Discovery Service** dynamically locates relevant modules, while the **Planning Engine** assembles these into ordered execution graphs. Finally, the **Execution Engine** runs each step inside a sandbox that enforces the permission policies defined in [`permission_systems.py`](https://github.com/davidkimai/context-engineering/blob/main/permission_systems.py).

### Core Components Overview

The repository organizes functionality into distinct modules that handle specific concerns:

| Component | Purpose | Source File |
|-----------|---------|-------------|
| **Tool discovery** | Dynamically locate, verify, and rank available tool modules at runtime. | [`06_tool_integrated_reasoning/toolkits/advanced_tool_system/tool_discovery.py`](https://github.com/davidkimai/context-engineering/blob/main/06_tool_integrated_reasoning/toolkits/advanced_tool_system/tool_discovery.py) |
| **Planning engine** | Assemble ordered, context-aware execution plans using discovered tools. | [`06_tool_integrated_reasoning/toolkits/advanced_tool_system/planning_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/06_tool_integrated_reasoning/toolkits/advanced_tool_system/planning_engine.py) |
| **Tool registry** | Central catalogue of safe tool definitions and input schemas. | [`06_tool_integrated_reasoning/toolkits/basic_function_calling/function_registry.py`](https://github.com/davidkimai/context-engineering/blob/main/06_tool_integrated_reasoning/toolkits/basic_function_calling/function_registry.py) |
| **Execution sandbox** | Isolate tool calls, enforce permission policies, and capture results. | [`06_tool_integrated_reasoning/toolkits/basic_function_calling/execution_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/06_tool_integrated_reasoning/toolkits/basic_function_calling/execution_engine.py) |

## Implementing Dynamic Tool Discovery

### The Tool Specification Contract

Every tool module must expose a `tool_spec` dictionary that defines its interface, security requirements, and metadata. This contract enables the discovery engine to load and validate tools without executing arbitrary code.

Create a new tool by defining the specification in your module:

```python

# 06_tool_integrated_reasoning/toolkits/advanced_tool_system/example_tool.py

tool_spec = {
    "name": "example_tool",
    "description": "Fetches and summarises a remote document.",
    "inputs": {
        "type": "object",
        "properties": {
            "url": {"type": "string", "format": "uri"}
        },
        "required": ["url"]
    },
    "outputs": {
        "type": "object",
        "properties": {
            "summary": {"type": "string"}
        }
    },
    "security": ["network"]
}

```

The **security** field lists required permissions that the execution engine checks against the agent's granted scopes before running the tool.

### Scanning and Loading Tool Modules

The discovery engine implements a filesystem scanner that walks the `toolkits/` directory, dynamically imports Python modules, and extracts their specifications safely.

Implement the discovery routine as follows:

```python

# tool_discovery.py

import importlib.util
import pathlib
from typing import List, Dict

TOOL_ROOT = pathlib.Path(__file__).parents[2] / "toolkits"

def _load_spec(module_path: pathlib.Path) -> Dict:
    """Safely load tool_spec from a module without polluting the global namespace."""
    spec = importlib.util.spec_from_file_location(module_path.stem, module_path)
    mod = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(mod)
    return getattr(mod, "tool_spec", {})

def discover_tools() -> List[Dict]:
    """Scan toolkits directory and return list of valid tool specifications."""
    tools = []
    for py_file in TOOL_ROOT.rglob("*.py"):
        spec = _load_spec(py_file)
        if spec:
            tools.append(spec)
    return tools

```

This implementation uses `importlib.util` to execute modules in isolated namespaces, preventing malicious code from affecting the discovery process itself.

## Building the Planning Engine for Complex Agents

### Strategy-Driven Plan Generation

The planning engine transforms discovered tools into executable sequences based on high-level strategies. Rather than hard-coding tool chains, the engine accepts a **strategy token** (such as `"guided_discovery"`) that determines how tools are ordered and parameterized.

The core planning logic resides in [`planning_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/planning_engine.py):

```python

# planning_engine.py

from typing import List, Dict

def plan_sequence(tools: List[Dict], strategy: str = "default") -> List[Dict]:
    """
    Generate an ordered execution plan based on the selected strategy.
    
    Args:
        tools: List of discovered tool specifications
        strategy: Planning strategy identifier ("default", "guided_discovery", etc.)
    
    Returns:
        List of (tool_name, parameters) dictionaries representing the execution plan
    """
    if strategy == "guided_discovery":
        # Inject reflection before execution to surface hidden sub-questions

        plan = [
            {"tool": "reflection_prompt", "parameters": {"prompt_type": "discovery"}},
            {"tool": "explanation_tool",   "parameters": {"complexity": "adaptive"}},
            {"tool": "assessment_tool",   "parameters": {"assessment_type": "formative"}},
            {"tool": "feedback_tool",     "parameters": {"feedback_type": "constructive"}},
        ]
    else:
        # Default: linear execution of all discovered tools

        plan = [{"tool": t["name"], "parameters": {}} for t in tools]
    
    return plan

```

### The Guided Discovery Pattern

The **guided discovery** strategy demonstrates how complex agents can implement meta-cognitive planning. Instead of immediately executing domain tools, the planner first invokes a `reflection_prompt` tool to analyze the problem structure.

This pattern appears in [`cognitive-tools/cognitive-architectures/architecture-examples.py`](https://github.com/davidkimai/context-engineering/blob/main/cognitive-tools/cognitive-architectures/architecture-examples.py) at lines 1910-1925, where the strategy injects a reflection step before the normal tool chain:

```python

# architecture-examples.py (lines 1910-1925)

def build_guided_sequence():
    """
    Constructs a tool sequence that begins with reflection to maximize
    context awareness before execution.
    """
    sequence = []
    
    # Step 1: Reflection to surface hidden requirements

    sequence.append({
        "tool": "reflection_prompt",
        "config": {"prompt_type": "discovery", "depth": "analytical"}
    })
    
    # Step 2-4: Domain-specific processing tools

    sequence.extend([
        {"tool": "explanation_tool", "config": {"complexity": "adaptive"}},
        {"tool": "assessment_tool", "config": {"assessment_type": "formative"}},
        {"tool": "feedback_tool", "config": {"feedback_type": "constructive"}}
    ])
    
    return sequence

```

This approach enables agents to adapt their planning based on intermediate reflections, making the system suitable for open-ended, complex problem domains where initial requirements are unclear.

## Safety and Execution Considerations

Before any planned tool executes, the system must verify permissions and isolate side effects. The [`execution_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/execution_engine.py) module implements a sandboxed runner that coordinates with [`permission_systems.py`](https://github.com/davidkimai/context-engineering/blob/main/permission_systems.py) to enforce security policies.

Key safety checks include:

- **Permission validation** – Matching the tool's `security` list (e.g., `["network"]`) against the agent's granted scopes before execution.
- **Resource quotas** – Enforcing CPU-time, memory, and I/O limits to prevent runaway processes.
- **Result sanitization** – Stripping secret-type fields from tool outputs before returning them to the planner or user.

This layered approach ensures that dynamically discovered tools cannot exceed their authority, even when the planning engine selects them automatically.

## Summary

- **Tool discovery** in `davidkimai/context-engineering` relies on a filesystem scanner that safely imports modules and extracts `tool_spec` dictionaries containing metadata, input schemas, and security requirements.
- **Planning engines** generate execution sequences based on strategies like `"guided_discovery"`, which can inject meta-cognitive steps (e.g., `reflection_prompt`) before domain tools.
- **Safety** is enforced through a permission system and sandboxed execution engine that validates tool capabilities against granted scopes before running any code.
- The modular architecture separates discovery, planning, and execution, enabling agents to self-configure at runtime without hard-coded dependencies.

## Frequently Asked Questions

### What is the tool specification contract in the context-engineering repository?

The **tool specification contract** requires every tool module to expose a top-level `tool_spec` dictionary containing `name`, `description`, `inputs` (JSON Schema), `outputs` (JSON Schema), and `security` (list of required permissions). This contract enables the discovery engine in [`tool_discovery.py`](https://github.com/davidkimai/context-engineering/blob/main/tool_discovery.py) to load and validate tools without executing arbitrary code, as seen in the [`example_tool.py`](https://github.com/davidkimai/context-engineering/blob/main/example_tool.py) implementation.

### How does the guided discovery strategy differ from default planning?

The **guided discovery** strategy injects a `reflection_prompt` tool at the beginning of the execution sequence to surface hidden sub-questions and requirements before invoking domain-specific tools. In contrast, the **default** strategy simply executes discovered tools in a linear sequence. The guided approach, implemented in [`planning_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/planning_engine.py) and demonstrated in [`architecture-examples.py`](https://github.com/davidkimai/context-engineering/blob/main/architecture-examples.py) (lines 1910-1925), enables meta-cognitive planning suitable for open-ended problems where initial requirements are unclear.

### What safety mechanisms prevent unauthorized tool execution?

The **execution engine** enforces three primary safety mechanisms: **permission validation**, which matches a tool's `security` list against the agent's granted scopes before execution; **resource quotas**, which limit CPU time, memory, and I/O to prevent runaway processes; and **result sanitization**, which strips sensitive fields from tool outputs before they reach the planner or user. These checks are coordinated between [`execution_engine.py`](https://github.com/davidkimai/context-engineering/blob/main/execution_engine.py) and [`permission_systems.py`](https://github.com/davidkimai/context-engineering/blob/main/permission_systems.py) in the repository.

### Can the discovery engine load tools from external packages?

Yes, the **discovery engine** can be extended to scan any Python package path, not just the local `toolkits/` directory. The `discover_tools()` function in [`tool_discovery.py`](https://github.com/davidkimai/context-engineering/blob/main/tool_discovery.py) uses `pathlib` to walk filesystem locations and `importlib.util` to safely import modules in isolated namespaces. By modifying the `TOOL_ROOT` constant or passing additional paths to the scanner, you can register tools from external pip-installed packages or remote repositories that follow the `tool_spec` contract.