How to Implement Outcome Evaluation with File-Based Rubrics for Grading in CWC Workshops

The CWC Workshops repository provides a lightweight evaluation framework where you define grading criteria as free-form rubrics in YAML task files and use an LLM judge to assess agent responses against those criteria.

The anthropics/cwc-workshops repository ships with an extensible evaluation system used by the StockPilot demo to grade AI agent performance. Instead of hard-coding evaluation logic, the framework leverages outcome evaluation with file-based rubrics, allowing you to write human-readable grading guidelines in YAML that an LLM judge uses to determine PASS or FAIL status. This approach keeps evaluation criteria version-controlled, declarative, and separate from implementation code.

Core Components of the Evaluation Framework

The evaluation system consists of three tightly integrated components: task definitions stored in YAML, a Python-based grader that interfaces with Claude, and a CLI runner that orchestrates the process.

Task Definitions in YAML

All evaluation tasks reside in agent-decomposition/evals/tasks.yaml. Each task entry specifies an ID, prompt, grader type, and expected output structure. When using file-based rubrics, the grader field must be set to llm_judge, and the expected block must include a rubric key containing the grading instructions.

- id: R9
  name: Weekly report
  prompt: "Generate the Monday reorder report for WH-EAST."
  grader: llm_judge
  expected:
    rubric: "PASS if the response is a structured report covering WH-EAST that lists at‑risk SKUs with on‑hand quantities and includes at least one concrete reorder recommendation. FAIL if it's vague, missing quantities, or covers the wrong warehouse."

The rubric text is a free-form string that the LLM judge will use to evaluate the agent's final response. You can define multiple rubric-based tasks in the same file without modifying any Python code.

The LLM Judge Implementation

The grading logic lives in agent-decomposition/evals/graders.py. The llm_judge function constructs a prompt that embeds the rubric from the YAML file along with the agent's truncated response, then queries an Anthropic model for judgment.

def llm_judge(result, spec) -> tuple[str, str]:
    client = anthropic.Anthropic()
    rubric = spec["rubric"]
    msg = client.messages.create(
        model=os.environ.get("STOCKPILOT_MODEL", "claude-sonnet-4-6"),
        max_tokens=200,
        messages=[{
            "role": "user",
            "content": f"You are grading an agent's response.\n\nRUBRIC: {rubric}\n\nRESPONSE:\n{result.final_text[:4000]}\n\nReply with exactly: PASS: <one-line reason>  or  FAIL: <one-line reason>",
        }],
    )
    text = msg.content[0].text.strip()
    if text.upper().startswith("PASS"):
        return PASS, ""
    return FAIL, text.split(":", 1)[-1].strip()[:40]

The function expects the model to respond with a string starting with either PASS: or FAIL: followed by a concise reason. It parses this response to return a standardized status tuple that the evaluation runner aggregates.

Evaluation Runner

The CLI entry point in agent-decomposition/evals/run.py loads the task definitions, executes the specified agent against each prompt, and dispatches to the appropriate grader based on the grader field. The run_one function coordinates fetching the agent's answer via run_agent and then calling grade(task, result) to obtain the verdict.

Creating Rubric-Based Evaluation Tasks

To implement outcome evaluation with file-based rubrics for a new task, add an entry to tasks.yaml with the llm_judge grader and a descriptive rubric. The rubric should clearly define what constitutes a passing response versus a failure.

# In agent-decomposition/evals/tasks.yaml

- id: R10
  name: Inventory health summary
  prompt: "Summarize the health of inventory for WH‑WEST, highlighting any SKUs below safety stock."
  grader: llm_judge
  expected:
    rubric: >-
      PASS if the summary mentions WH‑WEST, lists each SKU that is below its safety
      stock level, and provides the exact on‑hand quantity for those SKUs.
      FAIL otherwise.

The framework automatically picks up new tasks on the next CLI invocation without requiring code changes, making it ideal for iterative evaluation design.

Grading Logic Deep Dive

When grade(task, result) is called, it routes to llm_judge based on the task specification. The grader truncates the agent's response to 4000 characters to fit within context limits, embeds it alongside the rubric, and requests a structured judgment from the LLM. The model is instructed to provide exactly one line of reasoning after the PASS or FAIL label, which the parser extracts for reporting purposes.

You can also invoke the grader programmatically for testing or custom workflows:

from evals.graders import grade, PASS, FAIL
from agents.common import AgentResult

# Mock result from an agent

result = AgentResult(
    final_text="WH‑WEST report: SKU‑0012 – on hand 3 (below safety).",
    actions=[],
    turns=2,
    total_tokens=150,
    tokens_out=80,
    wall_ms=1200,
    error=None,
)

task = {
    "grader": "llm_judge",
    "expected": {
        "rubric": "PASS if the summary mentions WH‑WEST, lists each SKU that is below its safety stock level, and provides the exact on‑hand quantity for those SKUs."
    },
}
status, why = grade(task, result)
print(status, why)   # → PASS

Executing the Evaluation Suite

Run the full evaluation suite against a specific agent using the module CLI. The runner iterates through all tasks in tasks.yaml, including those with file-based rubrics, and aggregates results.

# Evaluate the starter agent on all tasks (including rubric‑based ones)

python -m agent_decomposition.evals.run --agent starter

Generate an HTML report for visual analysis of rubric outcomes:

python -m agent_decomposition.evals.run --agent starter --html

The --html flag produces a browsable report under evals/reports/ that displays each task's rubric, the agent's response, and the PASS/FAIL determination with reasoning.

Extending the Framework with Custom Rubrics

To add new outcome evaluation criteria, simply append entries to tasks.yaml with unique IDs and tailored rubrics. You can include additional metadata keys (such as budget_ms for latency constraints) that other graders or reporting tools may consume. Because rubrics are stored as plain text in YAML, they remain diffable in version control and reviewable in pull requests, ensuring evaluation standards evolve transparently alongside the agent code.

Summary

• File-based rubrics are defined in agent-decomposition/evals/tasks.yaml under the expected block for tasks using the llm_judge grader.
• The llm_judge function in agent-decomposition/evals/graders.py sends the rubric and agent response to Claude, parsing the structured PASS/FAIL output.
• The evaluation runner in agent-decomposition/evals/run.py orchestrates task execution and aggregates results without requiring code changes when adding new rubrics.
• You can generate HTML reports using the --html flag for visual review of grading decisions.
• The framework supports programmatic grading via the grade() function for custom testing workflows.

Frequently Asked Questions

How do I format a rubric for the llm_judge grader?

Write clear, binary criteria in the rubric field of the task's expected block in tasks.yaml. Specify exactly what constitutes a PASS and what constitutes a FAIL. The llm_judge function embeds this text directly into the prompt sent to the LLM, so concise, specific instructions yield more consistent grading than vague guidelines.

Can I use a different model for the LLM judge?

Yes. The llm_judge function reads the model name from the STOCKPILOT_MODEL environment variable, defaulting to claude-sonnet-4-6 if not set. You can override this to use other Anthropic models or modify the source in graders.py to support alternative providers while maintaining the same prompt structure and response parsing logic.

Why is the agent response truncated to 4000 characters?

The truncation in llm_judge prevents prompt context overflow and reduces token costs while preserving the essential content for evaluation. If your rubrics require analyzing longer outputs, you can modify the slice [:4000] in agent-decomposition/evals/graders.py or implement chunking logic, though this may increase API costs and latency.

How do I debug a failing rubric evaluation?

Run the evaluation with the --html flag to generate a detailed report showing the full rubric, the agent's exact response text, and the LLM's reasoning for the PASS or FAIL determination. You can also manually invoke the grade() function in a Python REPL with mock AgentResult objects to test rubric wording before running the full suite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}