# How to Implement Outcome Evaluation with File-Based Rubrics for Grading in CWC Workshops > Learn how to implement outcome evaluation with file-based rubrics for grading in CWC workshops. Use YAML rubrics and an LLM judge to assess agent responses effectively. - Repository: [Anthropic/cwc-workshops](https://github.com/anthropics/cwc-workshops) - Tags: tutorial - Published: 2026-07-18 --- **The CWC Workshops repository provides a lightweight evaluation framework where you define grading criteria as free-form rubrics in YAML task files and use an LLM judge to assess agent responses against those criteria.** The **anthropics/cwc-workshops** repository ships with an extensible evaluation system used by the StockPilot demo to grade AI agent performance. Instead of hard-coding evaluation logic, the framework leverages **outcome evaluation with file-based rubrics**, allowing you to write human-readable grading guidelines in YAML that an LLM judge uses to determine PASS or FAIL status. This approach keeps evaluation criteria version-controlled, declarative, and separate from implementation code. ## Core Components of the Evaluation Framework The evaluation system consists of three tightly integrated components: task definitions stored in YAML, a Python-based grader that interfaces with Claude, and a CLI runner that orchestrates the process. ### Task Definitions in YAML All evaluation tasks reside in [`agent-decomposition/evals/tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/tasks.yaml). Each task entry specifies an ID, prompt, grader type, and expected output structure. When using **file-based rubrics**, the `grader` field must be set to `llm_judge`, and the `expected` block must include a `rubric` key containing the grading instructions. ```yaml - id: R9 name: Weekly report prompt: "Generate the Monday reorder report for WH-EAST." grader: llm_judge expected: rubric: "PASS if the response is a structured report covering WH-EAST that lists at‑risk SKUs with on‑hand quantities and includes at least one concrete reorder recommendation. FAIL if it's vague, missing quantities, or covers the wrong warehouse." ``` The rubric text is a free-form string that the LLM judge will use to evaluate the agent's final response. You can define multiple rubric-based tasks in the same file without modifying any Python code. ### The LLM Judge Implementation The grading logic lives in [`agent-decomposition/evals/graders.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/graders.py). The `llm_judge` function constructs a prompt that embeds the rubric from the YAML file along with the agent's truncated response, then queries an Anthropic model for judgment. ```python def llm_judge(result, spec) -> tuple[str, str]: client = anthropic.Anthropic() rubric = spec["rubric"] msg = client.messages.create( model=os.environ.get("STOCKPILOT_MODEL", "claude-sonnet-4-6"), max_tokens=200, messages=[{ "role": "user", "content": f"You are grading an agent's response.\n\nRUBRIC: {rubric}\n\nRESPONSE:\n{result.final_text[:4000]}\n\nReply with exactly: PASS: or FAIL: ", }], ) text = msg.content[0].text.strip() if text.upper().startswith("PASS"): return PASS, "" return FAIL, text.split(":", 1)[-1].strip()[:40] ``` The function expects the model to respond with a string starting with either `PASS:` or `FAIL:` followed by a concise reason. It parses this response to return a standardized status tuple that the evaluation runner aggregates. ### Evaluation Runner The CLI entry point in [`agent-decomposition/evals/run.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/run.py) loads the task definitions, executes the specified agent against each prompt, and dispatches to the appropriate grader based on the `grader` field. The `run_one` function coordinates fetching the agent's answer via `run_agent` and then calling `grade(task, result)` to obtain the verdict. ## Creating Rubric-Based Evaluation Tasks To implement **outcome evaluation with file-based rubrics** for a new task, add an entry to [`tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/tasks.yaml) with the `llm_judge` grader and a descriptive rubric. The rubric should clearly define what constitutes a passing response versus a failure. ```yaml # In agent-decomposition/evals/tasks.yaml - id: R10 name: Inventory health summary prompt: "Summarize the health of inventory for WH‑WEST, highlighting any SKUs below safety stock." grader: llm_judge expected: rubric: >- PASS if the summary mentions WH‑WEST, lists each SKU that is below its safety stock level, and provides the exact on‑hand quantity for those SKUs. FAIL otherwise. ``` The framework automatically picks up new tasks on the next CLI invocation without requiring code changes, making it ideal for iterative evaluation design. ## Grading Logic Deep Dive When `grade(task, result)` is called, it routes to `llm_judge` based on the task specification. The grader truncates the agent's response to 4000 characters to fit within context limits, embeds it alongside the rubric, and requests a structured judgment from the LLM. The model is instructed to provide exactly one line of reasoning after the PASS or FAIL label, which the parser extracts for reporting purposes. You can also invoke the grader programmatically for testing or custom workflows: ```python from evals.graders import grade, PASS, FAIL from agents.common import AgentResult # Mock result from an agent result = AgentResult( final_text="WH‑WEST report: SKU‑0012 – on hand 3 (below safety).", actions=[], turns=2, total_tokens=150, tokens_out=80, wall_ms=1200, error=None, ) task = { "grader": "llm_judge", "expected": { "rubric": "PASS if the summary mentions WH‑WEST, lists each SKU that is below its safety stock level, and provides the exact on‑hand quantity for those SKUs." }, } status, why = grade(task, result) print(status, why) # → PASS ``` ## Executing the Evaluation Suite Run the full evaluation suite against a specific agent using the module CLI. The runner iterates through all tasks in [`tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/tasks.yaml), including those with file-based rubrics, and aggregates results. ```bash # Evaluate the starter agent on all tasks (including rubric‑based ones) python -m agent_decomposition.evals.run --agent starter ``` Generate an HTML report for visual analysis of rubric outcomes: ```bash python -m agent_decomposition.evals.run --agent starter --html ``` The `--html` flag produces a browsable report under `evals/reports/` that displays each task's rubric, the agent's response, and the PASS/FAIL determination with reasoning. ## Extending the Framework with Custom Rubrics To add new **outcome evaluation** criteria, simply append entries to [`tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/tasks.yaml) with unique IDs and tailored rubrics. You can include additional metadata keys (such as `budget_ms` for latency constraints) that other graders or reporting tools may consume. Because rubrics are stored as plain text in YAML, they remain diffable in version control and reviewable in pull requests, ensuring evaluation standards evolve transparently alongside the agent code. ## Summary - **File-based rubrics** are defined in [`agent-decomposition/evals/tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/tasks.yaml) under the `expected` block for tasks using the `llm_judge` grader. - The `llm_judge` function in [`agent-decomposition/evals/graders.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/graders.py) sends the rubric and agent response to Claude, parsing the structured PASS/FAIL output. - The evaluation runner in [`agent-decomposition/evals/run.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/run.py) orchestrates task execution and aggregates results without requiring code changes when adding new rubrics. - You can generate HTML reports using the `--html` flag for visual review of grading decisions. - The framework supports programmatic grading via the `grade()` function for custom testing workflows. ## Frequently Asked Questions ### How do I format a rubric for the llm_judge grader? Write clear, binary criteria in the `rubric` field of the task's `expected` block in [`tasks.yaml`](https://github.com/anthropics/cwc-workshops/blob/main/tasks.yaml). Specify exactly what constitutes a PASS and what constitutes a FAIL. The `llm_judge` function embeds this text directly into the prompt sent to the LLM, so concise, specific instructions yield more consistent grading than vague guidelines. ### Can I use a different model for the LLM judge? Yes. The `llm_judge` function reads the model name from the `STOCKPILOT_MODEL` environment variable, defaulting to `claude-sonnet-4-6` if not set. You can override this to use other Anthropic models or modify the source in [`graders.py`](https://github.com/anthropics/cwc-workshops/blob/main/graders.py) to support alternative providers while maintaining the same prompt structure and response parsing logic. ### Why is the agent response truncated to 4000 characters? The truncation in `llm_judge` prevents prompt context overflow and reduces token costs while preserving the essential content for evaluation. If your rubrics require analyzing longer outputs, you can modify the slice `[:4000]` in [`agent-decomposition/evals/graders.py`](https://github.com/anthropics/cwc-workshops/blob/main/agent-decomposition/evals/graders.py) or implement chunking logic, though this may increase API costs and latency. ### How do I debug a failing rubric evaluation? Run the evaluation with the `--html` flag to generate a detailed report showing the full rubric, the agent's exact response text, and the LLM's reasoning for the PASS or FAIL determination. You can also manually invoke the `grade()` function in a Python REPL with mock `AgentResult` objects to test rubric wording before running the full suite.