# How Outcomes Are Graded Using the Scorecard System in Research Desk

> Learn how outcomes are graded using the scorecard system in Research Desk. Discover concise task definitions and grading rubrics evaluated by a strict JSON schema validation.

- Repository: [Anthropic/cwc-workshops](https://github.com/anthropics/cwc-workshops)
- Tags: how-to-guide
- Published: 2026-07-18

---

**Outcomes in the Research Desk are concise task definitions paired with grading rubrics, and they are evaluated through a strict JSON schema validation system that produces a machine-readable scorecard.**

In the **anthropics/cwc-workshops** repository, the Research Desk component implements an outcome-driven workflow where each analyst session targets a specific deliverable. The system uses a rigorous **scorecard grading** mechanism to validate whether the AI analyst has met the defined criteria, transforming qualitative research tasks into structured, verifiable results.

## What Is an Outcome?

An **outcome** represents the "done" state for a research task. It combines a textual description of the expected result with a rubric that instructs the model on how to evaluate its own output. According to the source code in [`research-desk/src/lib/sessions.ts`](https://github.com/anthropics/cwc-workshops/blob/main/research-desk/src/lib/sessions.ts), outcomes are initiated via a `user.define_outcome` event that structures the session's objective and evaluation criteria.

When you define an outcome, you specify:
- A clear description of the expected deliverable (e.g., "Summarize the company's Q2 earnings and assess future growth")
- A rubric dictating valid values for enumerated fields like `guidance_tone` and `confidence`

## The Scorecard Grading Workflow

The **scorecard system** executes a four-phase grading pipeline that validates the analyst's output against the outcome requirements. This process is orchestrated in [`research-desk/src/lib/analysis.ts`](https://github.com/anthropics/cwc-workshops/blob/main/research-desk/src/lib/analysis.ts) and enforced by the schema definitions in [`research-desk/src/lib/scorecard.ts`](https://github.com/anthropics/cwc-workshops/blob/main/research-desk/src/lib/scorecard.ts).

### 1. Defining the Outcome

The workflow begins when the system sends a `user.define_outcome` event. This payload, handled in [`sessions.ts`](https://github.com/anthropics/cwc-workshops/blob/main/sessions.ts), establishes the target criteria the analyst must achieve. The rubric within the outcome definition constrains the possible values for fields such as guidance tone and confidence levels.

```typescript
// sessions.ts – outcome definition structure
//   type: "user.define_outcome",
//   outcome: {
//     description: "Summarize the company’s Q2 earnings and assess future growth.",
//     rubric: {
//       guidance_tone: ["positive","neutral","cautious","none"],
//       confidence: ["low","medium","high"]
//     }
//   }

```

### 2. Running the Analyst Session

During execution, the analyst processes the research task and generates a [`scorecard.json`](https://github.com/anthropics/cwc-workshops/blob/main/scorecard.json) file in the session's output folder. This JSON document contains the structured findings that claim to satisfy the outcome definition.

### 3. Validating the Scorecard

The `validateScorecard` function in [`scorecard.ts`](https://github.com/anthropics/cwc-workshops/blob/main/scorecard.ts) performs strict schema validation on the generated JSON. This function ensures:
- All required fields are present (e.g., `ticker`, `company_name`, `one_line_thesis`)
- String lengths are trimmed to caps and list items are within limits
- Enumerated fields match allowed values from `GUIDANCE_TONES` and `CONFIDENCE_LEVELS`

Any validation failures are collected in a `problems` array that surfaces specific schema violations to the user.

```typescript
import { validateScorecard, type Scorecard } from "./scorecard";

const parsed = JSON.parse(scorecardFile.bytes.toString("utf-8"));
const { scorecard, problems } = validateScorecard(parsed);

record.scorecard = scorecard;
record.status = scorecard ? "succeeded" : "failed";
record.problems = problems;

```

### 4. Recording the Grading Outcome

After validation, the orchestrator stores the scorecard or records the failure. The session record receives a status of `succeeded` or `failed` based on validation results. The UI renders the `one_line_thesis` and other structured fields from the validated scorecard, providing users with a clear assessment of how well the outcome was met.

## Scorecard Schema and Validation Rules

The **Scorecard** interface in [`research-desk/src/lib/scorecard.ts`](https://github.com/anthropics/cwc-workshops/blob/main/research-desk/src/lib/scorecard.ts) defines the machine-readable structure that validates grading outcomes. The schema enforces data types and enumerations to ensure consistency across analyst sessions.

```typescript
export interface Scorecard {
  ticker: string;
  company_name: string;
  filing_form: string;
  fiscal_period: string;
  filing_date: string;
  revenue_usd_m: number;
  revenue_yoy_pct: number;
  gross_margin_pct: number;
  operating_margin_pct: number;
  guidance_tone: string;          // must be one of GUIDANCE_TONES
  top_risks: string[];
  risk_factor_changes: string[];
  red_flags: string[];
  one_line_thesis: string;
  confidence: string;             // must be one of CONFIDENCE_LEVELS
  memory_note_path: string;
  embedded_instructions_flag: boolean;
  // optional fields …
}

```

The validation logic specifically guards against:
- Missing required financial metrics or company identifiers
- Invalid enumeration values outside the defined `GUIDANCE_TONES` or `CONFIDENCE_LEVELS` sets
- Excessively long strings or lists that exceed length caps

## Summary

- **Outcomes** are task definitions that combine a description with a grading rubric, initiated via `user.define_outcome` events in [`sessions.ts`](https://github.com/anthropics/cwc-workshops/blob/main/sessions.ts).
- The **scorecard system** validates analyst output through the `validateScorecard` function, which enforces a strict TypeScript schema against generated JSON.
- Validation failures populate a `problems` array, while successful validations produce a structured scorecard with fields like `one_line_thesis`, `guidance_tone`, and `confidence`.
- Session status is binary (`succeeded` or `failed`) based on whether the scorecard passes schema validation.

## Frequently Asked Questions

### What happens when a scorecard fails validation?

When `validateScorecard` detects schema violations—such as missing required fields or invalid enum values—it returns a `problems` array containing specific error details. The session status is set to `failed`, and these validation errors are surfaced to the user through the UI.

### Which values are valid for guidance_tone and confidence?

According to the scorecard schema in [`scorecard.ts`](https://github.com/anthropics/cwc-workshops/blob/main/scorecard.ts), the `guidance_tone` field must match one of the values defined in the `GUIDANCE_TONES` enumeration (typically `"positive"`, `"neutral"`, `"cautious"`, or `"none"`), while `confidence` must match a value from `CONFIDENCE_LEVELS` (such as `"low"`, `"medium"`, or `"high"`). The validator rejects any values outside these predefined sets.

### How does the system handle oversized content in scorecard fields?

The `validateScorecard` implementation automatically trims long strings to maximum lengths and caps the number of items in list fields (such as `top_risks` and `red_flags`). This prevents malformed or excessively verbose outputs from passing validation while preserving the core content.

### Where is the scorecard grading logic tested?

Unit tests for the validation logic reside in [`research-desk/tests/scorecard.test.ts`](https://github.com/anthropics/cwc-workshops/blob/main/research-desk/tests/scorecard.test.ts). These tests verify that the `validateScorecard` function correctly identifies valid scorecards, catches schema violations, and properly handles edge cases like empty fields or out-of-bounds enumeration values.