How the Quiz Structure Is Defined and Validated in AI Engineering From Scratch

The AI‑Engineering‑From‑Scratch curriculum enforces a strict JSON schema for every lesson quiz, requiring exactly six questions with a fixed stage distribution, and validates them automatically via Python audit scripts that emit L006 error codes and block CI pipelines on any violation.

The rohitg00/ai-engineering-from-scratch repository maintains assessment consistency across its open-source curriculum through a rigorous quiz structure definition and automated validation system. Every lesson must include a quiz.json file that conforms to a documented contract specifying field types, question counts, and answer formats. This article examines the schema specification in AGENTS.md, the validation logic in scripts/audit_lessons.py, and the CI enforcement mechanisms that ensure data integrity for the site generator.

The Quiz.json Schema Contract

The canonical schema resides in AGENTS.md under the "Lesson contract → quiz.json schema" section. This document defines the exact shape every quiz file must follow to be considered valid by the curriculum toolchain.

Required Top-Level Fields

Every quiz.json must be a valid JSON object containing three mandatory keys:

• lesson: A string matching the lesson directory slug
• title: A human-readable string describing the quiz
• questions: An array containing exactly six question objects

The questions array follows a strict distribution model designed to assess knowledge at different learning stages: one pre-assessment question, three check-in questions, and two post-assessment questions.

Question Object Structure

Each entry in the questions array must be an object with the following properties:

Field	Type	Constraints
stage	string	Must be one of: pre, check, post
question	string	Free-text prompt shown to learners
options	array	Exactly four string elements
correct	integer	Zero-based index (0-3) pointing to the correct option
explanation	string	Optional free-text providing rationale

The zero-based indexing for the correct field is critical—valid values are 0, 1, 2, or 3, corresponding to positions in the options array.

Automated Validation Pipeline

The repository enforces the quiz contract through two Python auditing scripts that run in CI, preventing malformed quizzes from reaching the main branch.

Lesson-Level Auditing

scripts/audit_lessons.py performs comprehensive validation on every lesson's quiz.json file. According to the source code, this script executes the following checks:

• Verifies the file contains valid JSON syntax
• Confirms the top-level object includes required keys (lesson, title, questions)
• Ensures questions is a non-empty array with exactly six entries
• Validates each question's stage value is one of the allowed enums (pre, check, post)
• Checks that options arrays contain exactly four string elements
• Confirms correct indices are integers within the range 0-3
• Enforces the required distribution: 1 pre, 3 check, 2 post questions

When violations are detected, the script emits L006 audit codes with descriptive messages. For example:

::error file=phases/04-computer-vision/01-image-fundamentals/quiz.json,line=23::L006 – quiz.json must contain exactly 6 questions; found 5
::error file=phases/04-computer-vision/01-image-fundamentals/quiz.json,line=45::L006 – question 3: `correct` index 4 out of range (must be 0‑3)

Certification Quiz Validation

For certification-level assessments, scripts/audit_certifications.py applies the same rigorous validation rules. This ensures consistency between lesson quizzes and final certification exams, maintaining uniform data structures that the site generator can reliably render.

CI Enforcement and Error Codes

The validation scripts run automatically via .github/workflows/curriculum.yml on every push and pull request. Any L006 error fails the build pipeline, blocking merges that contain malformed quizzes. This proactive enforcement guarantees that site/build.js—the static site generator—can safely assume all quiz data is well-formed when rendering lesson pages.

Example Quiz Structure

A compliant quiz.json file follows this exact pattern, as documented in AGENTS.md:

{
  "lesson": "01-image-fundamentals",
  "title": "Image Fundamentals Quiz",
  "questions": [
    {
      "stage": "pre",
      "question": "What does a pixel represent?",
      "options": ["A color value", "A sound", "A network packet", "A file"],
      "correct": 0,
      "explanation": "Pixels are the smallest addressable elements in an image."
    },
    {
      "stage": "check",
      "question": "Which filter reduces noise?",
      "options": ["Gaussian blur", "Edge detection", "Sharpen", "Histogram equalization"],
      "correct": 0
    },
    {
      "stage": "check",
      "question": "What is the purpose of a color channel?",
      "options": ["Store intensity for one primary color", "Store depth information", "Encode metadata", "Compress the image"],
      "correct": 0
    },
    {
      "stage": "check",
      "question": "Which format supports transparency?",
      "options": ["JPEG", "PNG", "BMP", "TIFF"],
      "correct": 1
    },
    {
      "stage": "post",
      "question": "How many bits per channel does a standard 8‑bit image use?",
      "options": ["4", "8", "16", "32"],
      "correct": 1
    },
    {
      "stage": "post",
      "question": "Which operation converts a color image to grayscale?",
      "options": ["Histogram equalization", "Median filtering", "Luminance weighting", "Thresholding"],
      "correct": 2
    }
  ]
}

Notice the strict adherence to six total questions, four options per question, zero-based correct indices, and the required 1-3-2 stage distribution.

Summary

• The AGENTS.md file defines the canonical quiz.json schema requiring six questions with specific field types and value constraints.
• Zero-based indexing (0-3) is enforced for the correct answer field, with exactly four options per question.
• Two Python scripts—scripts/audit_lessons.py and scripts/audit_certifications.py—perform automated validation emitting L006 error codes on violations.
• The CI pipeline defined in .github/workflows/curriculum.yml blocks merges containing invalid quiz structures.
• This validation ensures site/build.js can safely render quizzes without runtime data errors.

Frequently Asked Questions

What is the exact structure required for a quiz.json file?

Every quiz.json must contain a lesson string, title string, and questions array with exactly six objects. Each question needs a stage (pre/check/post), question text, four options, and a correct index (0-3). The six questions must follow a 1-3-2 distribution: one pre-assessment, three check-ins, and two post-assessment questions.

How does the CI pipeline validate quiz files?

The pipeline runs scripts/audit_lessons.py and scripts/audit_certifications.py on every pull request. These scripts parse each quiz.json to verify JSON validity, required fields, exact question counts, stage distributions, and answer index ranges. Violations generate L006 error annotations that fail the build.

What happens if a quiz.json file violates the schema?

The CI pipeline emits specific error messages with file paths and line numbers, such as "quiz.json must contain exactly 6 questions; found 5" or "correct index 4 out of range." These L006 errors block the merge, preventing malformed data from reaching the main branch and breaking the site generator.

Where is the quiz schema documented?

The authoritative schema definition lives in AGENTS.md under the "Lesson contract → quiz.json schema" section. This document specifies field types, constraints, and the required six-question structure that all automated validators enforce.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}