# How the Quiz Structure Is Defined and Validated in AI Engineering From Scratch

> Learn how AI engineering quizzes are defined and validated using a strict JSON schema and Python audit scripts. Discover the process and error codes used to ensure quiz integrity.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-09-11

---

**The AI‑Engineering‑From‑Scratch curriculum enforces a strict JSON schema for every lesson quiz, requiring exactly six questions with a fixed stage distribution, and validates them automatically via Python audit scripts that emit L006 error codes and block CI pipelines on any violation.**

The rohitg00/ai-engineering-from-scratch repository maintains assessment consistency across its open-source curriculum through a rigorous quiz structure definition and automated validation system. Every lesson must include a [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file that conforms to a documented contract specifying field types, question counts, and answer formats. This article examines the schema specification in **AGENTS.md**, the validation logic in **scripts/audit_lessons.py**, and the CI enforcement mechanisms that ensure data integrity for the site generator.

## The Quiz.json Schema Contract

The canonical schema resides in **AGENTS.md** under the "Lesson contract → quiz.json schema" section. This document defines the exact shape every quiz file must follow to be considered valid by the curriculum toolchain.

### Required Top-Level Fields

Every [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) must be a valid JSON object containing three mandatory keys:

- **`lesson`**: A string matching the lesson directory slug
- **`title`**: A human-readable string describing the quiz
- **`questions`**: An array containing exactly **six** question objects

The `questions` array follows a strict distribution model designed to assess knowledge at different learning stages: **one** *pre*-assessment question, **three** *check*-in questions, and **two** *post*-assessment questions.

### Question Object Structure

Each entry in the `questions` array must be an object with the following properties:

| Field | Type | Constraints |
|-------|------|-------------|
| `stage` | string | Must be one of: `pre`, `check`, `post` |
| `question` | string | Free-text prompt shown to learners |
| `options` | array | Exactly **four** string elements |
| `correct` | integer | **Zero-based** index (0-3) pointing to the correct option |
| `explanation` | string | Optional free-text providing rationale |

The zero-based indexing for the `correct` field is critical—valid values are 0, 1, 2, or 3, corresponding to positions in the `options` array.

## Automated Validation Pipeline

The repository enforces the quiz contract through two Python auditing scripts that run in CI, preventing malformed quizzes from reaching the main branch.

### Lesson-Level Auditing

**[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** performs comprehensive validation on every lesson's [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file. According to the source code, this script executes the following checks:

- Verifies the file contains valid JSON syntax
- Confirms the top-level object includes required keys (`lesson`, `title`, `questions`)
- Ensures `questions` is a non-empty array with exactly **six** entries
- Validates each question's `stage` value is one of the allowed enums (`pre`, `check`, `post`)
- Checks that `options` arrays contain exactly four string elements
- Confirms `correct` indices are integers within the range 0-3
- Enforces the required distribution: 1 pre, 3 check, 2 post questions

When violations are detected, the script emits **L006** audit codes with descriptive messages. For example:

```text
::error file=phases/04-computer-vision/01-image-fundamentals/quiz.json,line=23::L006 – quiz.json must contain exactly 6 questions; found 5
::error file=phases/04-computer-vision/01-image-fundamentals/quiz.json,line=45::L006 – question 3: `correct` index 4 out of range (must be 0‑3)

```

### Certification Quiz Validation

For certification-level assessments, **[`scripts/audit_certifications.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_certifications.py)** applies the same rigorous validation rules. This ensures consistency between lesson quizzes and final certification exams, maintaining uniform data structures that the site generator can reliably render.

### CI Enforcement and Error Codes

The validation scripts run automatically via **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)** on every push and pull request. Any L006 error fails the build pipeline, blocking merges that contain malformed quizzes. This proactive enforcement guarantees that **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)**—the static site generator—can safely assume all quiz data is well-formed when rendering lesson pages.

## Example Quiz Structure

A compliant [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file follows this exact pattern, as documented in **AGENTS.md**:

```json
{
  "lesson": "01-image-fundamentals",
  "title": "Image Fundamentals Quiz",
  "questions": [
    {
      "stage": "pre",
      "question": "What does a pixel represent?",
      "options": ["A color value", "A sound", "A network packet", "A file"],
      "correct": 0,
      "explanation": "Pixels are the smallest addressable elements in an image."
    },
    {
      "stage": "check",
      "question": "Which filter reduces noise?",
      "options": ["Gaussian blur", "Edge detection", "Sharpen", "Histogram equalization"],
      "correct": 0
    },
    {
      "stage": "check",
      "question": "What is the purpose of a color channel?",
      "options": ["Store intensity for one primary color", "Store depth information", "Encode metadata", "Compress the image"],
      "correct": 0
    },
    {
      "stage": "check",
      "question": "Which format supports transparency?",
      "options": ["JPEG", "PNG", "BMP", "TIFF"],
      "correct": 1
    },
    {
      "stage": "post",
      "question": "How many bits per channel does a standard 8‑bit image use?",
      "options": ["4", "8", "16", "32"],
      "correct": 1
    },
    {
      "stage": "post",
      "question": "Which operation converts a color image to grayscale?",
      "options": ["Histogram equalization", "Median filtering", "Luminance weighting", "Thresholding"],
      "correct": 2
    }
  ]
}

```

Notice the strict adherence to six total questions, four options per question, zero-based correct indices, and the required 1-3-2 stage distribution.

## Summary

- The **AGENTS.md** file defines the canonical [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) schema requiring six questions with specific field types and value constraints.
- **Zero-based indexing** (0-3) is enforced for the `correct` answer field, with exactly four options per question.
- Two Python scripts—**[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** and **[`scripts/audit_certifications.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_certifications.py)**—perform automated validation emitting **L006** error codes on violations.
- The CI pipeline defined in **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)** blocks merges containing invalid quiz structures.
- This validation ensures **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** can safely render quizzes without runtime data errors.

## Frequently Asked Questions

### What is the exact structure required for a quiz.json file?

Every [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) must contain a `lesson` string, `title` string, and `questions` array with exactly six objects. Each question needs a `stage` (pre/check/post), `question` text, four `options`, and a `correct` index (0-3). The six questions must follow a 1-3-2 distribution: one pre-assessment, three check-ins, and two post-assessment questions.

### How does the CI pipeline validate quiz files?

The pipeline runs **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** and **[`scripts/audit_certifications.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_certifications.py)** on every pull request. These scripts parse each [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) to verify JSON validity, required fields, exact question counts, stage distributions, and answer index ranges. Violations generate **L006** error annotations that fail the build.

### What happens if a quiz.json file violates the schema?

The CI pipeline emits specific error messages with file paths and line numbers, such as "`quiz.json must contain exactly 6 questions; found 5`" or "`correct` index 4 out of range." These **L006** errors block the merge, preventing malformed data from reaching the main branch and breaking the site generator.

### Where is the quiz schema documented?

The authoritative schema definition lives in **AGENTS.md** under the "Lesson contract → quiz.json schema" section. This document specifies field types, constraints, and the required six-question structure that all automated validators enforce.