# Claude Agent Evaluation with Decision Probes: A Technical Guide to the CWC Agent-Battle Harness

> Evaluate Claude Managed Agents with decision probes in the CWC agent-battle harness. This technical guide details using synthetic Minecraft states for rapid, rubric-based scoring.

- Repository: [Anthropic/cwc-workshops](https://github.com/anthropics/cwc-workshops)
- Tags: how-to-guide
- Published: 2026-07-20

---

**The agent-battle workshop in the `anthropics/cwc-workshops` repository provides a decision-probe harness that evaluates Claude Managed Agents using synthetic Minecraft states and rubric-based scoring, delivering feedback in approximately 10 seconds per probe.**

Claude agent evaluation with decision probes offers a deterministic alternative to lengthy game simulations. Instead of waiting five minutes for a full Minecraft episode to verify agent behavior, developers can use the lightweight probe harness to test specific decision points in isolation. This approach, implemented in the agent-battle workshop, lets you validate whether your Claude Managed Agent (CMA) makes optimal choices before committing to full-scale evaluation.

## What Are Decision Probes?

**Decision probes** are self-contained test cases that combine a synthetic game state with a scoring rubric to evaluate a single agent action. Each probe represents a critical scenario—such as mining diamond ore at the wrong depth or crafting without sufficient resources—and tests whether the agent selects the correct first tool call.

According to the source code in [`agent-battle/harness/probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/probes.py), a probe encapsulates:
- A unique identifier and human-readable title
- A synthetic `/state` dictionary representing the Minecraft environment
- A scoring callback that grades the agent's response
- Optional flags for capturing lookup tool calls

Because probes run in complete isolation with deterministic inputs, they eliminate the noise and variance inherent in full game simulations.

## Core Components of the Evaluation Harness

The probe evaluation system consists of four tightly-coupled components defined across the harness modules.

### The Probe Dataclass

The `Probe` dataclass in [`agent-battle/harness/probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/probes.py) serves as the container for each test case:

```python
@dataclass
class Probe:
    key: str                     # Unique identifier

    title: str                   # Human-readable description

    state: dict                  # Synthetic /state JSON

    score: Callable[[str, dict], tuple[str, str]]
    capture_lookup: bool = False # Whether to keep the lookup slot open

    preamble: str = ""           # Extra message lines for complex scenarios

```

Each instance bundles everything required for a single evaluation, making the system declarative and extensible.

### Synthetic State Builder

The `_state` helper function generates minimal, valid Minecraft state dictionaries with sensible defaults for position, health, inventory, and nearby blocks. Developers override specific fields to create targeted scenarios:

```python
_state(
    position={"x": 10, "y": -40, "z": 320},
    inventory=[
        {"name": "stone_pickaxe", "count": 1},
        {"name": "iron_ingot", "count": 3}
    ],
    equipped={"hand": "stone_pickaxe"},
    nearby_blocks=[{"name": "deepslate_diamond_ore", "distance": 3.2}]
)

```

This approach ensures consistency across probes while allowing precise control over test conditions.

### Scoring Callbacks

Scoring functions like `_score_iron_required` and `_score_depth` (defined in [`probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/probes.py)) evaluate the agent's chosen tool call against domain-specific logic. Each callback receives the tool name and arguments, then returns a tuple of `(grade, explanation)`:

- **✓ (Good)**: The agent selected the optimal action
- **⚠ (Warning)**: The action is acceptable but suboptimal  
- **✗ (Poor)**: The action is incorrect or dangerous

For example, the `iron_required` probe checks whether the agent places a crafting table before attempting to craft an iron pickaxe, or whether it incorrectly attempts to mine diamond ore with a stone pickaxe.

### The Harness Runner

The [`my_agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/my_agent.py) CLI entry point (invoked with `--eval`) iterates over the `PROBES` list, sends each synthetic state to the participant's CMA via the Claude SDK, and applies the probe's scoring rubric. Located in [`agent-battle/harness/my_agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/my_agent.py), this runner coordinates the low-level communication handled by [`agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent.py) and the tool wrappers defined in [`tools.py`](https://github.com/anthropics/cwc-workshops/blob/main/tools.py).

## How the Probe Evaluation Loop Works

The decision-probe methodology follows a four-step loop that executes in approximately 10 seconds per probe:

1. **Generate** a synthetic state describing the scenario you want to test using the `_state` builder.
2. **Run** the agent once against that state via the `run_one_action` helper in [`agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent.py), which captures only the first tool invocation.
3. **Score** the returned tool call using the probe-specific callback to determine if the agent recognized the critical decision.
4. **Iterate** across all probes in the `PROBES` list to obtain a quantitative health check of the agent's decision-making logic.

Because the harness isolates the *first* action, it specifically measures whether the agent **recognizes the most critical decision** in each scenario before proceeding with execution.

## Implementing Decision Probes in Code

Below is a self-contained implementation that mirrors the `--eval` CLI functionality. This snippet demonstrates how to run the probe suite against any CMA-compatible agent outside the command-line interface:

```python

# -------------------------------------------------

# Quick probe runner – usable outside the CLI

# -------------------------------------------------

import json
from pathlib import Path
from typing import List, Tuple

# Import the probe definitions

from agent_battle.harness.probes import PROBES, Probe
from agent_battle.harness.agent import run_one_action  # internal helper that talks to the CMA SDK

def evaluate_agent(agent_id: str) -> List[Tuple[str, str, str]]:
    """
    Runs every decision probe against ``agent_id`` and returns a list of
    (probe_key, grade, explanation) tuples.
    """
    results = []
    for probe in PROBES:
        # 1️⃣  Send the synthetic state and capture the first tool call

        tool_name, tool_args = run_one_action(agent_id, probe.state)

        # 2️⃣  Score the response using the probe‑specific callback

        grade, explanation = probe.score(tool_name, tool_args)

        results.append((probe.key, grade, explanation))
    return results

# -------------------------------------------------

# Example usage

# -------------------------------------------------

if __name__ == "__main__":
    # Replace with your deployed Managed Agent ID (or local sandbox ID)

    my_agent_id = "agent_01Hxxxxxx"
    table = evaluate_agent(my_agent_id)

    # Pretty‑print a markdown table

    print("| Probe | Grade | Explanation |")
    print("|---|---|---|")
    for key, grade, expl in table:
        print(f"| {key} | {grade} | {expl} |")

```

Place this script in the repository root and execute it with `uv run script.py` after installing dependencies via `uv sync`. The `run_one_action` function handles the Claude Managed-Agent SDK communication, while the probe's `score` method provides the domain-specific evaluation.

## Key Files and Architecture

| File | Role |
|---|---|
| [`agent-battle/harness/probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/probes.py) | Defines the `Probe` dataclass, synthetic state builder (`_state`), scoring callbacks (`_score_*`), and the `PROBES` collection |
| [`agent-battle/harness/my_agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/my_agent.py) | CLI entry point supporting `--eval` flag; iterates probes and prints results |
| [`agent-battle/harness/agent.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/agent.py) | Core SDK integration that sends synthetic states and extracts first tool calls |
| [`agent-battle/harness/tools.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/tools.py) | Thin wrappers around CMA tool calls including `lookup`, `mine_block`, and `equip` |
| [`agent-battle/harness/logging_.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/logging_.py) | Logging utilities for probe execution |

## Why Use Decision Probes for Claude Agent Evaluation?

Decision probes provide three distinct advantages over traditional full-game evaluation:

- **Rapid iteration cycles**: Each probe completes in ~10 seconds compared to 5-minute full simulations, enabling tight feedback loops during prompt engineering or skill development.
- **Deterministic regression detection**: Grades (`✓`, `⚠`, `✗`) provide quantifiable metrics that can be tracked across commits to catch decision quality regressions immediately.
- **Sandboxed scenario testing**: Synthetic states allow testing edge cases (like low health or specific inventory constraints) that may occur rarely in stochastic full-game runs.

When building Claude-based agents for the agent-battle competition—or any complex decision-making environment—integrating this probe suite into your CI pipeline ensures that system prompt tweaks and MCP tool bindings maintain optimal decision quality.

## Summary

- **Decision probes** in the `anthropics/cwc-workshops` repository provide fast, deterministic evaluation of Claude Managed Agents using synthetic Minecraft states.
- The `Probe` dataclass in [`probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/probes.py) encapsulates state definitions and scoring rubrics for isolated test cases.
- The evaluation harness captures only the **first tool call**, measuring whether the agent recognizes the critical decision in each scenario.
- Each probe runs in approximately **10 seconds**, making it practical to iterate on agent configurations without waiting for full game simulations.
- Scoring callbacks return standardized grades (`✓`, `⚠`, `✗`) that enable quantitative tracking of agent decision quality over time.

## Frequently Asked Questions

### How fast does each decision probe run?

Each decision probe completes in approximately 10 seconds. This represents a 30x speed improvement over full game simulations, which typically require five minutes per run. The speed comes from the harness only executing the first action against a synthetic state rather than simulating the entire Minecraft episode.

### What determines a passing grade on a probe?

The scoring callbacks in [`agent-battle/harness/probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-battle/harness/probes.py) evaluate both the tool name and arguments selected by the agent. A grade of **✓ (Good)** indicates the optimal action for the scenario, **⚠ (Warning)** indicates a suboptimal but acceptable choice, and **✗ (Poor)** indicates an incorrect or dangerous decision. The rubrics are scenario-specific; for example, the `iron_required` probe checks if the agent crafts an iron pickaxe before attempting to mine deepslate diamond ore.

### Can I add custom probes to the evaluation suite?

Yes. Create a new `Probe` instance with a unique key, synthetic state generated via the `_state` helper, and a custom scoring callback function that accepts `(tool_name, tool_args)` and returns a `(grade, explanation)` tuple. Append your probe to the `PROBES` list in [`probes.py`](https://github.com/anthropics/cwc-workshops/blob/main/probes.py), and the `--eval` CLI flag will automatically include it in the evaluation cycle.

### What is the difference between the probe harness and the full game simulation?

The probe harness tests **isolated decision points** using synthetic, deterministic states, while full game simulations run the agent in a live Minecraft environment for extended durations. Probes answer "Does the agent make the right first choice?" while full simulations answer "Does the agent survive and complete objectives over time?" Use probes for rapid iteration during development and full simulations for final validation.