quiz.json Schema and Validation Process in AI Engineering From Scratch

The AI Engineering From Scratch curriculum enforces a strict JSON schema for lesson quizzes defined in AGENTS.md and validated by scripts/audit_lessons.py, requiring exactly six questions with specific stage distributions and canonical keys.

The AI Engineering From Scratch repository by RohitG00 maintains consistency across 435 lessons through a rigorous quiz.json schema and validation process. Each lesson directory contains a quiz.json file that must conform to exact structural requirements to ensure compatibility with the learning platform's frontend. This standardized approach guarantees that every quiz renders correctly and provides a uniform experience for learners progressing through the curriculum.

Understanding the quiz.json Schema Structure

The schema definition resides in AGENTS.md (lines 88-102) and mandates a single JSON object with three mandatory top-level fields.

Top-Level Fields

Every quiz.json must contain these exact keys:

• ** lesson **: A string representing the directory slug (e.g., 01-intro-to-ml)
• ** title **: The human-readable lesson title presented to learners
• ** questions **: An array containing exactly six question objects

Question Object Requirements

Each question must use canonical keys only—legacy keys are strictly prohibited. The required structure includes:

Key	Type	Description
stage	string	One of pre, check, or post
question	string	The prompt text shown to learners
options	array[string]	Between 2 and 6 answer choices (inclusive)
correct	integer	Zero-based index pointing to the correct option
explanation	string	Rationale explaining why the answer is correct

Stage Distribution Rules

The editorial process requires a specific distribution across the six questions to ensure proper assessment coverage:

• 1 pre question: Assesses prior knowledge before lesson content
• 3 check questions: Tests comprehension during the lesson
• 2 post questions: Evaluates retention after content completion

Validation Process in audit_lessons.py

The automated validation logic is implemented in scripts/audit_lessons.py (lines 29-95), which runs on every pull request to enforce schema compliance and block invalid submissions.

JSON Parsing and Structure Checks

The validator first attempts to parse the file using json.loads. If parsing fails, it raises L006. The script accepts either a direct array of questions or an object containing a questions array—any other structure triggers L006.

Legacy Schema Detection

To prevent deprecated formats from entering the codebase, the script explicitly checks for legacy keys (q, choices, answer). Presence of any legacy key immediately raises L007, rejecting the file and prompting migration to canonical keys (question, options, correct).

Field Validation and Error Codes

For each question element, the validator performs granular checks with specific error codes:

• L006: Raised when canonical keys are missing or the JSON structure is invalid
• L008: Triggered when options is not an array of 2-6 strings
• L009: Raised when the correct integer points to an invalid index outside the options array range

Practical Examples

Minimal Valid quiz.json

The following demonstrates a compliant six-question structure adhering to all schema requirements including proper stage distribution:

{
  "lesson": "02-linear-regression",
  "title": "Linear Regression Basics",
  "questions": [
    {
      "stage": "pre",
      "question": "What is the purpose of a loss function?",
      "options": ["To evaluate model size", "To measure prediction error", "To generate random data", "To encrypt outputs"],
      "correct": 1,
      "explanation": "A loss function quantifies how far predictions are from true values."
    },
    {
      "stage": "check",
      "question": "Which method solves the normal equation directly?",
      "options": ["Gradient descent", "Stochastic gradient descent", "Closed‑form solution", "Monte‑Carlo sampling"],
      "correct": 2,
      "explanation": "The closed‑form solution computes the exact weights without iteration."
    },
    {
      "stage": "check",
      "question": "What does over‑fitting indicate?",
      "options": ["Model under‑performance on training data", "Excellent generalisation", "High variance", "Low bias"],
      "correct": 2,
      "explanation": "Over‑fitting shows the model captures noise, leading to high variance."
    },
    {
      "stage": "check",
      "question": "Which metric is appropriate for regression?",
      "options": ["Accuracy", "Precision", "Mean Squared Error", "F1‑Score"],
      "correct": 2,
      "explanation": "MSE measures average squared error across continuous predictions."
    },
    {
      "stage": "post",
      "question": "When should you add regularisation?",
      "options": ["When training loss is low", "When validation loss > training loss", "When dataset is tiny", "Never"],
      "correct": 1,
      "explanation": "A higher validation loss suggests over‑fitting, motivating regularisation."
    },
    {
      "stage": "post",
      "question": "What does the R² score close to 1 mean?",
      "options": ["Poor fit", "Perfect fit", "Random predictions", "Under‑fitting"],
      "correct": 1,
      "explanation": "R² near 1 indicates the model explains most variance in the data."
    }
  ]
}

Common Validation Errors

Using legacy keys like q instead of question triggers immediate rejection with error code L007:

{
  "lesson": "03-legacy-example",
  "title": "Legacy Schema Demo",
  "questions": [
    {
      "stage": "pre",
      "q": "What is a typo?",
      "choices": ["A grammar mistake", "A missing semicolon"],
      "answer": 0,
      "explanation": "Legacy keys will be rejected."
    }
  ]
}

Running python scripts/audit_lessons.py on this file produces:

L007: question[0] uses legacy schema keys ['answer', 'choices', 'q'] (canonical: ['correct', 'explanation', 'options', 'question', 'stage'])

Summary

• The quiz.json schema requires exactly six questions with canonical keys (stage, question, options, correct, explanation) formally defined in AGENTS.md
• scripts/audit_lessons.py enforces validation through specific error codes: L006 (structure/keys), L007 (legacy keys), L008 (options count), and L009 (index bounds)
• Legacy keys including q, choices, and answer are strictly prohibited and raise L007 errors during CI
• Each quiz must follow the stage distribution: 1 pre, 3 check, and 2 post questions to meet editorial standards
• The validation process runs automatically on every PR, blocking merges that contain invalid quiz structures

Frequently Asked Questions

What happens if my quiz.json uses legacy keys like "q" or "choices"?

The validator in scripts/audit_lessons.py detects legacy keys during the CI pipeline and raises error code L007. This error identifies the specific legacy keys found and lists the required canonical alternatives, forcing contributors to update their JSON structure before the PR can merge.

How many questions must each quiz contain?

While the validation script only guarantees that the questions array is non-empty and well-formed, the editorial standard defined in AGENTS.md requires exactly six questions per lesson. This specific count ensures consistent pacing and assessment coverage across all 435 lessons in the curriculum.

What are the valid values for the stage field?

The stage field accepts exactly three string values: pre for pre-assessment questions, check for comprehension checks during the lesson, and post for retention testing after content completion. The standard distribution requires one pre question, three check questions, and two post questions per quiz.

Where is the quiz.json schema formally defined?

The authoritative schema definition lives in AGENTS.md at lines 88-102, which documents the exact structure, required keys, and data types according to the source code. The enforcement mechanism resides in scripts/audit_lessons.py (lines 29-95), which implements the linting logic that validates files against this specification during continuous integration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}