# How to Design Agent Skill Orchestration and Active Tool Discovery

> Learn agent skill orchestration with state-machine workflows and role-based tool permissions. Discover active tool discovery for runtime expansion via semantic search when needed.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-08-17

---

**Agent skill orchestration guides LLMs through predefined state-machine workflows with role-based tool permissions, while active tool discovery enables runtime expansion of capabilities via semantic search when the current toolset is insufficient.**

The `bojieli/ai-agent-book` repository demonstrates production-ready patterns for building autonomous agents that can follow structured multi-role workflows and dynamically discover missing capabilities. This implementation combines rigid state-machine orchestration with flexible runtime tool discovery to create systems that behave like human teams—delegating tasks to specialists and requesting expertise on demand.

## Skill Orchestration: State-Machine Workflows with Tool Gating

Skill orchestration treats agent execution as a directed workflow through predefined roles (triage → research → coding → data_analysis → writing). Each skill operates within a sandboxed environment with a fixed system prompt and an explicit whitelist of authorized tools.

### Core Components

The orchestration layer resides in three primary files:

- **[`skill_orchestrator.py`](https://github.com/bojieli/ai-agent-book/blob/main/skill_orchestrator.py)** – Contains the `SkillOrchestrator` class that manages state transitions, enforces permission boundaries, and aggregates performance metrics.
- **[`roles.py`](https://github.com/bojieli/ai-agent-book/blob/main/roles.py)** – Defines skill specifications including system prompts and the `SKILL_TOOLS` mapping that authorizes specific tool sets per role.
- **[`tools.py`](https://github.com/bojieli/ai-agent-book/blob/main/tools.py)** – Hosts the global tool registry (`TOOL_IMPLEMENTATIONS` and `TOOL_SCHEMAS`) that serves as the source of truth for available capabilities.

### How Skill Orchestration Works

The `SkillOrchestrator` class implements a five-stage enforcement mechanism:

1. **System Prompt Construction** – A static `SKILL_SYSTEM_PROMPT` is built once during initialization and attached to every LLM request via `_messages_for_api`.
2. **Skill Loading via Tool Call** – The model must explicitly invoke `load_skill(name="triage")` to begin. The orchestrator validates the request, loads the markdown documentation from `SKILL_ROOT`, and caches it in `_skill_cache`.
3. **Tool Gating** – After loading, only tools listed in `SKILL_TOOLS[skill]` are exposed to the model. Unauthorized tool calls return a policy-rejection message without executing.
4. **History and Activity Logging** – Every assistant message, tool call, and final response is appended to `self.history`, while a lightweight `self.activity` log tracks which role performed which action for audit trails.
5. **Metrics Aggregation** – The `summary()` method computes token usage, cache-hit statistics, and step counts, producing reproducible performance reports.

```python
from openai import OpenAI
from chapter10.multi_role_transfer.skill_orchestrator import SkillOrchestrator

client = OpenAI()
agent = SkillOrchestrator(
    client=client,
    model="gpt-5.6-luna",
    max_steps=15,
    verbose=False,
)

# The orchestrator automatically enforces skill-loading order and tool permissions

final_answer = agent.run("请帮我分析 2023 年苹果公司的财务报表并写一段结论。")
print(final_answer)
print(agent.summary())  # token usage, cache hits, step counts

```

## Active Tool Discovery: Runtime Capability Expansion

Active tool discovery allows an LLM to request missing specialist tools when the baseline catalog cannot satisfy a task requirement. The system queries a semantic index of all available tool schemas, injects the top-k matches into the conversation, and dynamically expands the usable tool set.

### Experiment Infrastructure

The discovery mechanism relies on components from `chapter4/active-tool-discovery/`:

- **[`run_exact_experiment.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_exact_experiment.py)** – The experiment driver that initializes the baseline catalog, manages the discovery loop, and verifies final receipts.
- **`LocalEmbeddingIndex`** – A dense-vector index built from all MCP tool schemas that enables semantic similarity search.
- **`DISCOVER_SCHEMA`** – The JSON schema definition for the `discover_tools` function that the model can invoke.
- **Validation Helpers** – `parse_action`, `mcp_receipt`, and `grade_plan` handle action validation, execution logging, and accuracy computation.

### The Discovery Workflow

The active discovery process follows six distinct phases:

1. **Baseline Initialization** – The experiment begins with a restricted catalog (`BASE_TOOL_NAMES`) containing only generic utilities.
2. **Model Turn with Status** – The LLM receives a system prompt describing the task and current available tools, displayed in a status bar.
3. **Discovery Request** – When the model selects `action="discover_tools"`, the orchestrator queries `LocalEmbeddingIndex.search()` to retrieve the five most semantically similar schemas for the described need.
4. **Schema Injection** – Discovered schemas are rendered as JSON, hashed, and appended to the conversation history as user messages. The `available` tool set updates to include the new schema names.
5. **Specialist Tool Execution** – The model may now call newly exposed tools (e.g., `arxiv_search`, `github_list_contributors`). The runtime executes real MCP calls, records receipts via `mcp_receipt`, and verifies observations are substantive.
6. **Verification and Acceptance** – `derive_acceptance` checks that every required capability slot has a valid receipt, validates token counts, and confirms the campaign meets published acceptance gates.

```python
import asyncio
from chapter4.active_tool_discovery.run_exact_experiment import run

# Run control vs treatment experiments with dynamic tool discovery

records = asyncio.run(run(campaign_id="demo-2024"))
summary = safe_summary(records)  # aggregates accuracy & completion stats

print(summary["treatment"]["mean_tool_selection_accuracy"])

```

## Shared Architectural Principles

Both patterns leverage common infrastructure design decisions that ensure reliability and auditability.

### Shared Conversation History

All roles and tools operate on the same `history` list, ensuring every subsequent step sees the full conversational context without information silos.

### Explicit Tool Schema Exposure

The model only receives JSON schemas for tools explicitly authorized for the current skill or discovered via the semantic index. This prevents accidental leakage of privileged capabilities not relevant to the active task.

### Deterministic Auditing

Every LLM call, tool invocation, and role transfer is logged with timestamps, token counts, and `sha256_bytes` hash-chains. This enables reproducible verification through the `derive_acceptance` protocol.

### Graceful Failure Handling

When policy violations occur—such as calling an unauthorized tool or attempting invalid role transfers—the orchestrator returns structured error messages and logs the event. This allows the LLM to recover without crashing the pipeline.

## Summary

- **Skill orchestration** enforces deterministic workflows through `SkillOrchestrator`, using `load_skill()` calls and `SKILL_TOOLS` whitelists to gate tool access by role.
- **Active tool discovery** enables autonomous capability expansion via `LocalEmbeddingIndex` semantic search, injecting discovered schemas into the conversation history at runtime.
- Both patterns share **unified conversation history**, **explicit schema exposure**, and **deterministic auditing** with hash-chained receipts.
- The implementation in `bojieli/ai-agent-book` provides production-ready examples in [`skill_orchestrator.py`](https://github.com/bojieli/ai-agent-book/blob/main/skill_orchestrator.py) and [`run_exact_experiment.py`](https://github.com/bojieli/ai-agent-book/blob/main/run_exact_experiment.py).

## Frequently Asked Questions

### What is the difference between skill orchestration and active tool discovery?

Skill orchestration manages deterministic, state-machine workflows where the agent transitions through predefined roles (triage, research, coding) with fixed tool permissions. Active tool discovery is a dynamic mechanism that allows the agent to request and integrate new tools at runtime when the current catalog proves insufficient for a specific task.

### How does the orchestrator prevent unauthorized tool usage?

The `SkillOrchestrator` maintains a `SKILL_TOOLS` mapping that defines authorized tools per skill. After a skill is loaded via `load_skill()`, the orchestrator filters the global `TOOL_SCHEMAS` registry, exposing only whitelisted tools to the model. Attempts to call unauthorized tools return policy-rejection messages without execution.

### What happens when the LLM requests a tool that isn't in the current catalog?

When the model invokes `action="discover_tools"`, the system queries the `LocalEmbeddingIndex` to perform semantic similarity search against all available MCP schemas. The top-k matches are injected into the conversation history as JSON schemas, and the `available` tool set updates dynamically, allowing immediate use of the discovered tools in subsequent turns.

### How does the system ensure reproducibility and auditing?

Every operation generates structured logs with timestamps, token counts, and `sha256_bytes` hashes. The `derive_acceptance` function verifies that all required capability slots have valid receipts, checks that dynamic token counts match expectations, and validates the campaign against published acceptance gates, creating a cryptographically verifiable audit trail.