How to Implement Scorecard Validation for Agent Output Quality

Use the validateScorecard function in research-desk/src/lib/scorecard.ts to enforce strict schema compliance on agent outputs, returning either a sanitized Scorecard object or a detailed problems array when validation fails.

The anthropics/cwc-workshops repository demonstrates a robust pattern for implementing scorecard validation for agent output quality, ensuring that AI-generated financial analysis sessions emit predictable data structures. This validation layer acts as a critical gate between raw agent outputs and downstream consumers such as CSV exporters, synthesis engines, and analytics dashboards.

The Core Validation Function

The validation logic lives in research-desk/src/lib/scorecard.ts within the validateScorecard function. This utility enforces three fundamental guarantees on incoming payloads:

1. Known fields only – The function accepts required fields as mandatory, permits optional fields, and silently strips any extraneous keys.
2. Typed and bounded values – Strings truncate to MAX_SHORT_TEXT (300 characters), numeric fields must be finite numbers, booleans remain strictly boolean, and list fields cap at MAX_LIST_ITEMS (5 entries).
3. Enumerated constraints – Specific fields must match predefined allowed values.

According to the source code at lines 75–76, the function first guards against invalid payload types by verifying the input is a non-array object before proceeding with field-level validation.

Schema Constraints and Type Enforcement

The validation engine performs field-by-field cleaning through a loop over REQUIRED_FIELDS and OPTIONAL_FIELDS (lines 81–112). This process applies several defensive transformations:

• String truncation – Any text field exceeding 300 characters is sliced to prevent storage overflow.
• List capping – Arrays containing more than 5 items are truncated, with each remaining entry also subject to length limits.
• Type coercion protection – Numeric fields reject non-finite values (NaN, Infinity), ensuring downstream mathematical operations remain safe.

If a required field remains absent after the cleaning pass, the function immediately aborts and returns scorecard: null alongside a descriptive problems array (lines 21–23).

Enumerated Value Validation

The scorecard schema restricts specific categorical fields to enumerated values defined at lines 14–19. The validation logic uses array inclusion checks to enforce these constraints:

• guidance_tone must be one of: ["positive", "neutral", "cautious", "none"]
• confidence must be one of: ["low", "medium", "high"]

When an agent emits an unrecognized value for either field, validateScorecard captures the violation in the problems array and rejects the entire payload, preventing invalid categorical data from polluting analytics datasets.

Handling Validation Results

The function returns a discriminated union pattern that simplifies error handling in TypeScript:

const { scorecard, problems } = validateScorecard(rawPayload);

if (scorecard) {
  // Proceed with downstream processing
  forwardToSynthesis(scorecard);
} else {
  // Handle validation failures
  throw new Error(`Scorecard validation failed: ${problems.join("; ")}`);
}

This pattern ensures that only conforming data reaches CSV rendering functions like scorecardsToCsv, while providing granular feedback for debugging agent prompt engineering issues.

Testing the Validation Logic

The test suite in research-desk/tests/scorecard.test.ts (lines 28–51) codifies the expected behavior through comprehensive scenarios:

• Accepting fully populated, correct scorecards
• Rejecting payloads with missing required fields
• Flagging invalid enum values and type mismatches
• Enforcing list-length caps and string truncation limits

These tests serve as executable documentation for the validation contract, ensuring that schema changes maintain backward compatibility or explicitly update the test expectations.

Integration Patterns

CSV Export Pipeline

Once validated, scorecards integrate seamlessly with export utilities:

import { scorecardsToCsv } from "research-desk/src/lib/scorecard";

if (scorecard) {
  const csv = scorecardsToCsv([scorecard]);
  // Returns header row plus single data row
  console.log(csv);
}

Agent Pipeline Error Handling

For production agent orchestration, wrap validation in a retry-aware handler:

function processAgentOutput(raw: unknown) {
  const { scorecard, problems } = validateScorecard(raw);
  if (!scorecard) {
    // Surface to orchestrator for retry logic or incident logging
    throw new Error(`Validation failed: ${problems.join("; ")}`);
  }
  return scorecard;
}

Summary

• Validate early – Use validateScorecard from research-desk/src/lib/scorecard.ts immediately after agent output generation to catch schema violations before downstream processing.
• Enforce boundaries – Leverage built-in constants (MAX_SHORT_TEXT, MAX_LIST_ITEMS) to prevent data overflow and ensure consistent field sizes.
• Handle failures explicitly – Check for scorecard: null and inspect the problems array to debug agent prompt issues or data quality problems.
• Test thoroughly – Reference research-desk/tests/scorecard.test.ts for validation scenarios covering enums, types, and list constraints.

Frequently Asked Questions

What happens if the agent output contains extra fields not defined in the schema?

The validateScorecard function ignores any keys not listed in REQUIRED_FIELDS or OPTIONAL_FIELDS, effectively stripping extraneous data during the cleaning pass at lines 81–112. This prevents schema pollution while maintaining forward compatibility.

How does the validation handle list fields that exceed the maximum length?

Lists are truncated to MAX_LIST_ITEMS (5 entries), with each retained entry also subject to MAX_SHORT_TEXT truncation (300 characters). This occurs during the field-by-field loop in research-desk/src/lib/scorecard.ts, ensuring storage and display constraints remain satisfied.

Can the validation function handle non-object inputs gracefully?

Yes. At lines 75–76, the function checks whether the payload is a non-array object using typeof checks. If the input is an array, null, or primitive, validation immediately fails and returns scorecard: null with an appropriate error message in the problems array.

What are the valid values for guidance_tone and confidence fields?

According to the constants defined at lines 14–19, guidance_tone accepts "positive", "neutral", "cautious", or "none", while confidence accepts "low", "medium", or "high". Any other values trigger validation failures captured in the returned problems array.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}