How Outcomes Are Graded Using the Scorecard System in Research Desk

Outcomes in the Research Desk are concise task definitions paired with grading rubrics, and they are evaluated through a strict JSON schema validation system that produces a machine-readable scorecard.

In the anthropics/cwc-workshops repository, the Research Desk component implements an outcome-driven workflow where each analyst session targets a specific deliverable. The system uses a rigorous scorecard grading mechanism to validate whether the AI analyst has met the defined criteria, transforming qualitative research tasks into structured, verifiable results.

What Is an Outcome?

An outcome represents the "done" state for a research task. It combines a textual description of the expected result with a rubric that instructs the model on how to evaluate its own output. According to the source code in research-desk/src/lib/sessions.ts, outcomes are initiated via a user.define_outcome event that structures the session's objective and evaluation criteria.

When you define an outcome, you specify:

• A clear description of the expected deliverable (e.g., "Summarize the company's Q2 earnings and assess future growth")
• A rubric dictating valid values for enumerated fields like guidance_tone and confidence

The Scorecard Grading Workflow

The scorecard system executes a four-phase grading pipeline that validates the analyst's output against the outcome requirements. This process is orchestrated in research-desk/src/lib/analysis.ts and enforced by the schema definitions in research-desk/src/lib/scorecard.ts.

1. Defining the Outcome

The workflow begins when the system sends a user.define_outcome event. This payload, handled in sessions.ts, establishes the target criteria the analyst must achieve. The rubric within the outcome definition constrains the possible values for fields such as guidance tone and confidence levels.

// sessions.ts – outcome definition structure
//   type: "user.define_outcome",
//   outcome: {
//     description: "Summarize the company’s Q2 earnings and assess future growth.",
//     rubric: {
//       guidance_tone: ["positive","neutral","cautious","none"],
//       confidence: ["low","medium","high"]
//     }
//   }

2. Running the Analyst Session

During execution, the analyst processes the research task and generates a scorecard.json file in the session's output folder. This JSON document contains the structured findings that claim to satisfy the outcome definition.

3. Validating the Scorecard

The validateScorecard function in scorecard.ts performs strict schema validation on the generated JSON. This function ensures:

• All required fields are present (e.g., ticker, company_name, one_line_thesis)
• String lengths are trimmed to caps and list items are within limits
• Enumerated fields match allowed values from GUIDANCE_TONES and CONFIDENCE_LEVELS

Any validation failures are collected in a problems array that surfaces specific schema violations to the user.

import { validateScorecard, type Scorecard } from "./scorecard";

const parsed = JSON.parse(scorecardFile.bytes.toString("utf-8"));
const { scorecard, problems } = validateScorecard(parsed);

record.scorecard = scorecard;
record.status = scorecard ? "succeeded" : "failed";
record.problems = problems;

4. Recording the Grading Outcome

After validation, the orchestrator stores the scorecard or records the failure. The session record receives a status of succeeded or failed based on validation results. The UI renders the one_line_thesis and other structured fields from the validated scorecard, providing users with a clear assessment of how well the outcome was met.

Scorecard Schema and Validation Rules

The Scorecard interface in research-desk/src/lib/scorecard.ts defines the machine-readable structure that validates grading outcomes. The schema enforces data types and enumerations to ensure consistency across analyst sessions.

export interface Scorecard {
  ticker: string;
  company_name: string;
  filing_form: string;
  fiscal_period: string;
  filing_date: string;
  revenue_usd_m: number;
  revenue_yoy_pct: number;
  gross_margin_pct: number;
  operating_margin_pct: number;
  guidance_tone: string;          // must be one of GUIDANCE_TONES
  top_risks: string[];
  risk_factor_changes: string[];
  red_flags: string[];
  one_line_thesis: string;
  confidence: string;             // must be one of CONFIDENCE_LEVELS
  memory_note_path: string;
  embedded_instructions_flag: boolean;
  // optional fields …
}

The validation logic specifically guards against:

• Missing required financial metrics or company identifiers
• Invalid enumeration values outside the defined GUIDANCE_TONES or CONFIDENCE_LEVELS sets
• Excessively long strings or lists that exceed length caps

Summary

• Outcomes are task definitions that combine a description with a grading rubric, initiated via user.define_outcome events in sessions.ts.
• The scorecard system validates analyst output through the validateScorecard function, which enforces a strict TypeScript schema against generated JSON.
• Validation failures populate a problems array, while successful validations produce a structured scorecard with fields like one_line_thesis, guidance_tone, and confidence.
• Session status is binary (succeeded or failed) based on whether the scorecard passes schema validation.

Frequently Asked Questions

What happens when a scorecard fails validation?

When validateScorecard detects schema violations—such as missing required fields or invalid enum values—it returns a problems array containing specific error details. The session status is set to failed, and these validation errors are surfaced to the user through the UI.

Which values are valid for guidance_tone and confidence?

According to the scorecard schema in scorecard.ts, the guidance_tone field must match one of the values defined in the GUIDANCE_TONES enumeration (typically "positive", "neutral", "cautious", or "none"), while confidence must match a value from CONFIDENCE_LEVELS (such as "low", "medium", or "high"). The validator rejects any values outside these predefined sets.

How does the system handle oversized content in scorecard fields?

The validateScorecard implementation automatically trims long strings to maximum lengths and caps the number of items in list fields (such as top_risks and red_flags). This prevents malformed or excessively verbose outputs from passing validation while preserving the core content.

Where is the scorecard grading logic tested?

Unit tests for the validation logic reside in research-desk/tests/scorecard.test.ts. These tests verify that the validateScorecard function correctly identifies valid scorecards, catches schema violations, and properly handles edge cases like empty fields or out-of-bounds enumeration values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}