# How quiz.json Schema Validation Works in AI Engineering From Scratch

> Learn how AI Engineering From Scratch validates quiz.json schema using scripts to ensure structure, key names, and data integrity. Explore robust AI development practices.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-22

---

**The AI Engineering From Scratch curriculum enforces a strict JSON schema for every lesson's [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file using the [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) script, which validates structure, key names, and data integrity across five canonical fields.**

The `ai-engineering-from-scratch` repository maintains curriculum quality through automated validation that runs in CI and locally. Every lesson must include a [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file following a precise schema defined in the audit script. This ensures all quizzes are machine-readable, uniformly structured, and free of legacy formats before merging into the main branch.

## The Audit Script Architecture

Validation logic resides in **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)**, specifically within the `check_quiz` function. This script serves as the single source of truth for quiz schema compliance, processing each lesson directory to verify that [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) exists and conforms to the canonical structure.

The script defines three critical constants that drive validation decisions:

- **`CANONICAL_QUIZ_KEYS`** – The required set of exactly five keys: `stage`, `question`, `options`, `correct`, `explanation` (lines 30-31)
- **`LEGACY_QUIZ_KEYS`** – Deprecated keys that trigger migration warnings: `q`, `choices`, `answer` (lines 31-32)
- **`MIN_OPTIONS`** / **`MAX_OPTIONS`** – Integer bounds (2 and 6) restricting the `options` array length (lines 34-35)

## Step-by-Step Validation Process

The `check_quiz` function executes a nine-point inspection pipeline, collecting violations as `Issue` objects that are reported at the end of the audit run (lines 124-138).

### Parse and Load

First, the script attempts to read and parse the file using `json.loads` on UTF-8 encoded content. Any JSON syntax error or encoding issue immediately raises **`L006`** (lines 33-38).

### Topology Checks

The validator accepts two top-level shapes:

1. A direct list (array of question objects)
2. A dictionary containing a `questions` array

If the parsed data matches neither pattern, or if the `questions` array is empty or missing, the script raises **`L006`** (lines 39-53).

### Per-Question Object Validation

Each entry in the questions array undergoes granular inspection starting at line 53.

**Legacy Key Detection**

If any question contains deprecated keys (`q`, `choices`, `answer`), the script raises **`L007`** and points to the legacy schema (lines 57-66).

**Canonical Key Requirements**

Every question must contain exactly the five keys defined in `CANONICAL_QUIZ_KEYS`. Missing keys generate **`L006`** violations (lines 66-74).

**Options Array Constraints**

The `options` field must be a list with length between `MIN_OPTIONS` (2) and `MAX_OPTIONS` (6). Violations raise **`L008`** (lines 76-84).

**Correct Index Validation**

The `correct` field must be an integer that indexes a valid position within the `options` array (0 ≤ correct < len(options)). Out-of-bounds or non-integer values trigger **`L009`** (lines 86-93).

## Schema Constraints and Constants

The validation enforces rigid structural rules derived from the constants defined at the top of [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py):

| Constant | Value | Purpose |
|----------|-------|---------|
| `CANONICAL_QUIZ_KEYS` | `{'stage', 'question', 'options', 'correct', 'explanation'}` | Required keys per question |
| `LEGACY_QUIZ_KEYS` | `{'q', 'choices', 'answer'}` | Forbidden legacy keys |
| `MIN_OPTIONS` | `2` | Minimum number of answer choices |
| `MAX_OPTIONS` | `6` | Maximum number of answer choices |

## Practical Examples

### Valid Quiz Structure

A compliant [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) contains a `questions` array with objects using the canonical five keys:

```json
{
  "questions": [
    {
      "stage": "pre",
      "question": "What does AI stand for?",
      "options": ["Artificial Intelligence", "Automated Interface"],
      "correct": 0,
      "explanation": "AI is short for Artificial Intelligence."
    },
    {
      "stage": "check",
      "question": "Which of these is a type of neural network?",
      "options": ["CNN", "RNN", "Both", "None"],
      "correct": 2,
      "explanation": "Both CNN and RNN are neural network types."
    }
  ]
}

```

Running `python scripts/audit_lessons.py` against this file produces no issues.

### Common Validation Errors

**Insufficient Options**

```json
{
  "questions": [
    {
      "stage": "pre",
      "question": "Bad example",
      "options": ["Only one"],
      "correct": 0,
      "explanation": "Option count too low"
    }
  ]
}

```

This triggers: `[L008] …/quiz.json: question[0] options length must be 2..6 (got 1)`

**Legacy Schema Usage**

```json
{
  "questions": [
    { "q": "Legacy key", "choices": ["A", "B"], "answer": 1 }
  ]
}

```

This triggers: `[L007] …/quiz.json: question[0] uses legacy schema keys ['answer', 'choices', 'q'] (canonical: ['correct', 'explanation', 'options', 'question', 'stage'])`

## Summary

- The **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** script serves as the single validation authority for all [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) files in the repository.
- **Five canonical keys** are required: `stage`, `question`, `options`, `correct`, and `explanation`.
- **Legacy keys** (`q`, `choices`, `answer`) are forbidden and raise `L007` violations.
- **Options arrays** must contain between 2 and 6 items; violations raise `L008`.
- The **`correct`** field must be a valid integer index into the `options` array; invalid indices raise `L009`.
- Validation runs automatically in CI and can be executed locally with `python scripts/audit_lessons.py`.

## Frequently Asked Questions

### What happens if my quiz.json file is missing one of the required keys?

The audit script raises an **`L006`** error and identifies which canonical keys are missing. Every question object must contain exactly the five keys defined in `CANONICAL_QUIZ_KEYS`: `stage`, `question`, `options`, `correct`, and `explanation`.

### Can I use a simple array instead of a questions object at the top level?

Yes. The validator accepts either a direct list of question objects or a dictionary containing a `questions` array. Both formats are valid, but the array must be non-empty to pass validation.

### What is the maximum number of options allowed per question?

The schema allows between **2 and 6 options** per question. This is enforced by the `MIN_OPTIONS` (2) and `MAX_OPTIONS` (6) constants in [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py). Arrays with fewer than 2 or more than 6 items trigger **`L008`**.

### How do I run the validation locally before submitting a pull request?

Execute `python scripts/audit_lessons.py` from the repository root. The script checks all lessons and reports any schema violations, including legacy key usage, missing fields, and invalid option counts, allowing you to fix issues before CI runs.