How quiz.json Schema Validation Works in AI Engineering From Scratch

The AI Engineering From Scratch curriculum enforces a strict JSON schema for every lesson's quiz.json file using the scripts/audit_lessons.py script, which validates structure, key names, and data integrity across five canonical fields.

The ai-engineering-from-scratch repository maintains curriculum quality through automated validation that runs in CI and locally. Every lesson must include a quiz.json file following a precise schema defined in the audit script. This ensures all quizzes are machine-readable, uniformly structured, and free of legacy formats before merging into the main branch.

The Audit Script Architecture

Validation logic resides in scripts/audit_lessons.py, specifically within the check_quiz function. This script serves as the single source of truth for quiz schema compliance, processing each lesson directory to verify that quiz.json exists and conforms to the canonical structure.

The script defines three critical constants that drive validation decisions:

• CANONICAL_QUIZ_KEYS – The required set of exactly five keys: stage, question, options, correct, explanation (lines 30-31)
• LEGACY_QUIZ_KEYS – Deprecated keys that trigger migration warnings: q, choices, answer (lines 31-32)
• MIN_OPTIONS / MAX_OPTIONS – Integer bounds (2 and 6) restricting the options array length (lines 34-35)

Step-by-Step Validation Process

The check_quiz function executes a nine-point inspection pipeline, collecting violations as Issue objects that are reported at the end of the audit run (lines 124-138).

Parse and Load

First, the script attempts to read and parse the file using json.loads on UTF-8 encoded content. Any JSON syntax error or encoding issue immediately raises L006 (lines 33-38).

Topology Checks

The validator accepts two top-level shapes:

1. A direct list (array of question objects)
2. A dictionary containing a questions array

If the parsed data matches neither pattern, or if the questions array is empty or missing, the script raises L006 (lines 39-53).

Per-Question Object Validation

Each entry in the questions array undergoes granular inspection starting at line 53.

Legacy Key Detection

If any question contains deprecated keys (q, choices, answer), the script raises L007 and points to the legacy schema (lines 57-66).

Canonical Key Requirements

Every question must contain exactly the five keys defined in CANONICAL_QUIZ_KEYS. Missing keys generate L006 violations (lines 66-74).

Options Array Constraints

The options field must be a list with length between MIN_OPTIONS (2) and MAX_OPTIONS (6). Violations raise L008 (lines 76-84).

Correct Index Validation

The correct field must be an integer that indexes a valid position within the options array (0 ≤ correct < len(options)). Out-of-bounds or non-integer values trigger L009 (lines 86-93).

Schema Constraints and Constants

The validation enforces rigid structural rules derived from the constants defined at the top of audit_lessons.py:

Constant	Value	Purpose
CANONICAL_QUIZ_KEYS	{'stage', 'question', 'options', 'correct', 'explanation'}	Required keys per question
LEGACY_QUIZ_KEYS	{'q', 'choices', 'answer'}	Forbidden legacy keys
MIN_OPTIONS	2	Minimum number of answer choices
MAX_OPTIONS	6	Maximum number of answer choices

Practical Examples

Valid Quiz Structure

A compliant quiz.json contains a questions array with objects using the canonical five keys:

{
  "questions": [
    {
      "stage": "pre",
      "question": "What does AI stand for?",
      "options": ["Artificial Intelligence", "Automated Interface"],
      "correct": 0,
      "explanation": "AI is short for Artificial Intelligence."
    },
    {
      "stage": "check",
      "question": "Which of these is a type of neural network?",
      "options": ["CNN", "RNN", "Both", "None"],
      "correct": 2,
      "explanation": "Both CNN and RNN are neural network types."
    }
  ]
}

Running python scripts/audit_lessons.py against this file produces no issues.

Common Validation Errors

Insufficient Options

{
  "questions": [
    {
      "stage": "pre",
      "question": "Bad example",
      "options": ["Only one"],
      "correct": 0,
      "explanation": "Option count too low"
    }
  ]
}

This triggers: [L008] …/quiz.json: question[0] options length must be 2..6 (got 1)

Legacy Schema Usage

{
  "questions": [
    { "q": "Legacy key", "choices": ["A", "B"], "answer": 1 }
  ]
}

This triggers: [L007] …/quiz.json: question[0] uses legacy schema keys ['answer', 'choices', 'q'] (canonical: ['correct', 'explanation', 'options', 'question', 'stage'])

Summary

• The scripts/audit_lessons.py script serves as the single validation authority for all quiz.json files in the repository.
• Five canonical keys are required: stage, question, options, correct, and explanation.
• Legacy keys (q, choices, answer) are forbidden and raise L007 violations.
• Options arrays must contain between 2 and 6 items; violations raise L008.
• The correct field must be a valid integer index into the options array; invalid indices raise L009.
• Validation runs automatically in CI and can be executed locally with python scripts/audit_lessons.py.

Frequently Asked Questions

What happens if my quiz.json file is missing one of the required keys?

The audit script raises an L006 error and identifies which canonical keys are missing. Every question object must contain exactly the five keys defined in CANONICAL_QUIZ_KEYS: stage, question, options, correct, and explanation.

Can I use a simple array instead of a questions object at the top level?

Yes. The validator accepts either a direct list of question objects or a dictionary containing a questions array. Both formats are valid, but the array must be non-empty to pass validation.

What is the maximum number of options allowed per question?

The schema allows between 2 and 6 options per question. This is enforced by the MIN_OPTIONS (2) and MAX_OPTIONS (6) constants in audit_lessons.py. Arrays with fewer than 2 or more than 6 items trigger L008.

How do I run the validation locally before submitting a pull request?

Execute python scripts/audit_lessons.py from the repository root. The script checks all lessons and reports any schema violations, including legacy key usage, missing fields, and invalid option counts, allowing you to fix issues before CI runs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}