Claude Agent Evaluation with Decision Probes: A Technical Guide to the CWC Agent-Battle Harness

The agent-battle workshop in the anthropics/cwc-workshops repository provides a decision-probe harness that evaluates Claude Managed Agents using synthetic Minecraft states and rubric-based scoring, delivering feedback in approximately 10 seconds per probe.

Claude agent evaluation with decision probes offers a deterministic alternative to lengthy game simulations. Instead of waiting five minutes for a full Minecraft episode to verify agent behavior, developers can use the lightweight probe harness to test specific decision points in isolation. This approach, implemented in the agent-battle workshop, lets you validate whether your Claude Managed Agent (CMA) makes optimal choices before committing to full-scale evaluation.

What Are Decision Probes?

Decision probes are self-contained test cases that combine a synthetic game state with a scoring rubric to evaluate a single agent action. Each probe represents a critical scenario—such as mining diamond ore at the wrong depth or crafting without sufficient resources—and tests whether the agent selects the correct first tool call.

According to the source code in agent-battle/harness/probes.py, a probe encapsulates:

• A unique identifier and human-readable title
• A synthetic /state dictionary representing the Minecraft environment
• A scoring callback that grades the agent's response
• Optional flags for capturing lookup tool calls

Because probes run in complete isolation with deterministic inputs, they eliminate the noise and variance inherent in full game simulations.

Core Components of the Evaluation Harness

The probe evaluation system consists of four tightly-coupled components defined across the harness modules.

The Probe Dataclass

The Probe dataclass in agent-battle/harness/probes.py serves as the container for each test case:

@dataclass
class Probe:
    key: str                     # Unique identifier

    title: str                   # Human-readable description

    state: dict                  # Synthetic /state JSON

    score: Callable[[str, dict], tuple[str, str]]
    capture_lookup: bool = False # Whether to keep the lookup slot open

    preamble: str = ""           # Extra message lines for complex scenarios

Each instance bundles everything required for a single evaluation, making the system declarative and extensible.

Synthetic State Builder

The _state helper function generates minimal, valid Minecraft state dictionaries with sensible defaults for position, health, inventory, and nearby blocks. Developers override specific fields to create targeted scenarios:

_state(
    position={"x": 10, "y": -40, "z": 320},
    inventory=[
        {"name": "stone_pickaxe", "count": 1},
        {"name": "iron_ingot", "count": 3}
    ],
    equipped={"hand": "stone_pickaxe"},
    nearby_blocks=[{"name": "deepslate_diamond_ore", "distance": 3.2}]
)

This approach ensures consistency across probes while allowing precise control over test conditions.

Scoring Callbacks

Scoring functions like _score_iron_required and _score_depth (defined in probes.py) evaluate the agent's chosen tool call against domain-specific logic. Each callback receives the tool name and arguments, then returns a tuple of (grade, explanation):

• ✓ (Good): The agent selected the optimal action
• ⚠ (Warning): The action is acceptable but suboptimal
• ✗ (Poor): The action is incorrect or dangerous

For example, the iron_required probe checks whether the agent places a crafting table before attempting to craft an iron pickaxe, or whether it incorrectly attempts to mine diamond ore with a stone pickaxe.

The Harness Runner

The my_agent.py CLI entry point (invoked with --eval) iterates over the PROBES list, sends each synthetic state to the participant's CMA via the Claude SDK, and applies the probe's scoring rubric. Located in agent-battle/harness/my_agent.py, this runner coordinates the low-level communication handled by agent.py and the tool wrappers defined in tools.py.

How the Probe Evaluation Loop Works

The decision-probe methodology follows a four-step loop that executes in approximately 10 seconds per probe:

1. Generate a synthetic state describing the scenario you want to test using the _state builder.
2. Run the agent once against that state via the run_one_action helper in agent.py, which captures only the first tool invocation.
3. Score the returned tool call using the probe-specific callback to determine if the agent recognized the critical decision.
4. Iterate across all probes in the PROBES list to obtain a quantitative health check of the agent's decision-making logic.

Because the harness isolates the first action, it specifically measures whether the agent recognizes the most critical decision in each scenario before proceeding with execution.

Implementing Decision Probes in Code

Below is a self-contained implementation that mirrors the --eval CLI functionality. This snippet demonstrates how to run the probe suite against any CMA-compatible agent outside the command-line interface:

# -------------------------------------------------

# Quick probe runner – usable outside the CLI

# -------------------------------------------------

import json
from pathlib import Path
from typing import List, Tuple

# Import the probe definitions

from agent_battle.harness.probes import PROBES, Probe
from agent_battle.harness.agent import run_one_action  # internal helper that talks to the CMA SDK

def evaluate_agent(agent_id: str) -> List[Tuple[str, str, str]]:
    """
    Runs every decision probe against ``agent_id`` and returns a list of
    (probe_key, grade, explanation) tuples.
    """
    results = []
    for probe in PROBES:
        # 1️⃣  Send the synthetic state and capture the first tool call

        tool_name, tool_args = run_one_action(agent_id, probe.state)

        # 2️⃣  Score the response using the probe‑specific callback

        grade, explanation = probe.score(tool_name, tool_args)

        results.append((probe.key, grade, explanation))
    return results

# -------------------------------------------------

# Example usage

# -------------------------------------------------

if __name__ == "__main__":
    # Replace with your deployed Managed Agent ID (or local sandbox ID)

    my_agent_id = "agent_01Hxxxxxx"
    table = evaluate_agent(my_agent_id)

    # Pretty‑print a markdown table

    print("| Probe | Grade | Explanation |")
    print("|---|---|---|")
    for key, grade, expl in table:
        print(f"| {key} | {grade} | {expl} |")

Place this script in the repository root and execute it with uv run script.py after installing dependencies via uv sync. The run_one_action function handles the Claude Managed-Agent SDK communication, while the probe's score method provides the domain-specific evaluation.

Key Files and Architecture

File	Role
agent-battle/harness/probes.py	Defines the Probe dataclass, synthetic state builder (_state), scoring callbacks (_score_*), and the PROBES collection
agent-battle/harness/my_agent.py	CLI entry point supporting --eval flag; iterates probes and prints results
agent-battle/harness/agent.py	Core SDK integration that sends synthetic states and extracts first tool calls
agent-battle/harness/tools.py	Thin wrappers around CMA tool calls including lookup, mine_block, and equip
agent-battle/harness/logging_.py	Logging utilities for probe execution

Why Use Decision Probes for Claude Agent Evaluation?

Decision probes provide three distinct advantages over traditional full-game evaluation:

• Rapid iteration cycles: Each probe completes in ~10 seconds compared to 5-minute full simulations, enabling tight feedback loops during prompt engineering or skill development.
• Deterministic regression detection: Grades (✓, ⚠, ✗) provide quantifiable metrics that can be tracked across commits to catch decision quality regressions immediately.
• Sandboxed scenario testing: Synthetic states allow testing edge cases (like low health or specific inventory constraints) that may occur rarely in stochastic full-game runs.

When building Claude-based agents for the agent-battle competition—or any complex decision-making environment—integrating this probe suite into your CI pipeline ensures that system prompt tweaks and MCP tool bindings maintain optimal decision quality.

Summary

• Decision probes in the anthropics/cwc-workshops repository provide fast, deterministic evaluation of Claude Managed Agents using synthetic Minecraft states.
• The Probe dataclass in probes.py encapsulates state definitions and scoring rubrics for isolated test cases.
• The evaluation harness captures only the first tool call, measuring whether the agent recognizes the critical decision in each scenario.
• Each probe runs in approximately 10 seconds, making it practical to iterate on agent configurations without waiting for full game simulations.
• Scoring callbacks return standardized grades (✓, ⚠, ✗) that enable quantitative tracking of agent decision quality over time.

Frequently Asked Questions

How fast does each decision probe run?

Each decision probe completes in approximately 10 seconds. This represents a 30x speed improvement over full game simulations, which typically require five minutes per run. The speed comes from the harness only executing the first action against a synthetic state rather than simulating the entire Minecraft episode.

What determines a passing grade on a probe?

The scoring callbacks in agent-battle/harness/probes.py evaluate both the tool name and arguments selected by the agent. A grade of ✓ (Good) indicates the optimal action for the scenario, ⚠ (Warning) indicates a suboptimal but acceptable choice, and ✗ (Poor) indicates an incorrect or dangerous decision. The rubrics are scenario-specific; for example, the iron_required probe checks if the agent crafts an iron pickaxe before attempting to mine deepslate diamond ore.

Can I add custom probes to the evaluation suite?

Yes. Create a new Probe instance with a unique key, synthetic state generated via the _state helper, and a custom scoring callback function that accepts (tool_name, tool_args) and returns a (grade, explanation) tuple. Append your probe to the PROBES list in probes.py, and the --eval CLI flag will automatically include it in the evaluation cycle.

What is the difference between the probe harness and the full game simulation?

The probe harness tests isolated decision points using synthetic, deterministic states, while full game simulations run the agent in a live Minecraft environment for extended durations. Probes answer "Does the agent make the right first choice?" while full simulations answer "Does the agent survive and complete objectives over time?" Use probes for rapid iteration during development and full simulations for final validation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}