How Schema Validation for quiz.json Files Works in AI-Engineering-From-Scratch

The AI-Engineering-From-Scratch repository validates quiz.json files through a centralized audit script located at scripts/audit_lessons.py, which enforces canonical schema rules, rejects legacy key formats, and validates option arrays and correct answer indices.

The ai-engineering-from-scratch curriculum maintains strict data integrity through automated schema validation for quiz.json files. Located in the scripts/audit_lessons.py utility, this validation system ensures every lesson's quiz adheres to a canonical structure before passing continuous integration checks. The audit process validates JSON parsing, top-level data structures, per-question field requirements, and logical consistency across all question data.

The Audit Entry Point: scripts/audit_lessons.py

The scripts/audit_lessons.py file serves as the central validation driver for the entire curriculum. According to the source code, this script runs as part of the "audit" step that checks all lessons for compliance with curriculum rules. When the script encounters a quiz.json file in any lesson directory (typically located under phases/*/*/quiz.json), it triggers a comprehensive validation pipeline that reports specific error codes (L006-L009) when violations occur.

JSON Parsing and Top-Level Structure Validation

Parsing and Decoding Checks

The validation begins with strict JSON parsing. In audit_lessons.py lines 33-38, the script attempts to read and decode the file content using json.loads(). If the file contains malformed JSON, syntax errors, or encoding issues, the audit immediately reports an L006 issue and halts processing for that specific lesson.

Acceptable Root Schema Types

Following successful parsing, lines 39-45 in audit_lessons.py validate the top-level data structure. The schema accepts two valid formats:

• Direct array format: A JSON list where each element is a question object
• Wrapped object format: A JSON dictionary containing a non-empty "questions" key that holds the question array

Any other structure triggers an L006 issue, ensuring consistent data shapes across the curriculum.

Canonical Quiz Schema Definition

The expected schema is defined through two critical constants declared in lines 30-33 of audit_lessons.py:

CANONICAL_QUIZ_KEYS = {"stage", "question", "options", "correct", "explanation"}
LEGACY_QUIZ_KEYS = {"q", "choices", "answer"}
MIN_OPTIONS, MAX_OPTIONS = 2, 6

These constants establish the contract for valid quiz data. The canonical keys represent the current standard, while legacy keys trigger deprecation warnings. The options boundaries ensure every question contains between 2 and 6 possible answers.

Per-Question Validation Rules

The core validation logic spans lines 53-94 in audit_lessons.py, iterating through each question in the array and applying four distinct checks:

Legacy Key Detection (L007)

If any question contains deprecated keys (q, choices, or answer), the audit reports an L007 issue. This enforcement prevents drift toward outdated schema formats and ensures all content uses the modern canonical structure.

Required Field Validation (L006)

Every question must contain all five canonical keys: stage, question, options, correct, and explanation. The validation computes the set difference between CANONICAL_QUIZ_KEYS and the actual question keys. Missing fields generate an L006 issue with specific details about which keys are absent.

Options Array Constraints (L008)

The options field must be a list with a length between MIN_OPTIONS (2) and MAX_OPTIONS (6). Violations of this range produce an L008 issue, ensuring consistent user experience across all quiz questions.

Correct Answer Index Verification (L009)

The correct field must be an integer that points to a valid zero-based index within the options array. If the value is negative or exceeds the length of the options list, the audit raises an L009 issue. This prevents runtime errors where the correct answer points to a non-existent option.

Error Codes and CI Integration

The validation system uses a structured error code taxonomy:

• L006: JSON parsing failures, invalid top-level structures, or missing required keys
• L007: Usage of deprecated legacy keys
• L008: Options array length violations (fewer than 2 or more than 6 items)
• L009: Invalid correct answer indices

When any check fails, the script reports the specific error code, lesson path, and detailed diagnostic message, then aborts the CI check for that lesson. This prevents corrupted or malformed quiz data from merging into the main branch.

Summary

• Validation Location: All quiz.json validation occurs in scripts/audit_lessons.py
• Structure Requirements: Files must contain either a question array or an object with a "questions" key
• Canonical Fields: Valid questions require stage, question, options, correct, and explanation
• Legacy Migration: Deprecated keys (q, choices, answer) trigger L007 warnings
• Option Constraints: Questions must have 2-6 options (L008) with a valid index in correct (L009)
• Error Reporting: The system uses codes L006-L009 to categorize specific validation failures

Frequently Asked Questions

What happens if a quiz.json file contains invalid JSON?

The audit script attempts to parse the file using json.loads() in lines 33-38 of audit_lessons.py. If parsing fails due to malformed syntax or encoding errors, the system immediately reports an L006 issue and stops processing that specific lesson, preventing the invalid file from passing CI checks.

What is the difference between canonical and legacy quiz keys?

Canonical keys represent the current schema standard: stage, question, options, correct, and explanation. Legacy keys (q, choices, answer) are deprecated formats from earlier versions of the curriculum. If audit_lessons.py detects any legacy keys in a question, it raises an L007 issue to enforce migration to the modern schema.

How many options must a quiz question have?

Each question must contain between 2 and 6 options, as defined by the MIN_OPTIONS and MAX_OPTIONS constants (set to 2 and 6 respectively) in lines 30-33 of audit_lessons.py. Violations of these bounds generate an L008 error during the audit process.

Where is the quiz validation logic located?

The primary validation logic resides in scripts/audit_lessons.py, specifically in lines 53-94, which contain the per-question validation loop. This script checks JSON structure, key presence, options array length, and correct answer indices for every lesson's quiz.json file located under the phases/ directory structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}