# quiz.json Schema and Validation Process in AI Engineering From Scratch

> Understand the quiz.json schema and validation process for AI Engineering From Scratch. Learn how lesson quizzes are structured and validated with strict rules for questions and stages.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-09-04

---

**The AI Engineering From Scratch curriculum enforces a strict JSON schema for lesson quizzes defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) and validated by [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py), requiring exactly six questions with specific stage distributions and canonical keys.**

The AI Engineering From Scratch repository by RohitG00 maintains consistency across 435 lessons through a rigorous **quiz.json schema and validation process**. Each lesson directory contains a [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) file that must conform to exact structural requirements to ensure compatibility with the learning platform's frontend. This standardized approach guarantees that every quiz renders correctly and provides a uniform experience for learners progressing through the curriculum.

## Understanding the quiz.json Schema Structure

The schema definition resides in **AGENTS.md** (lines 88-102) and mandates a single JSON object with three mandatory top-level fields.

### Top-Level Fields

Every [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) must contain these exact keys:

- ** `lesson` **: A string representing the directory slug (e.g., `01-intro-to-ml`)
- ** `title` **: The human-readable lesson title presented to learners
- ** `questions` **: An array containing exactly six question objects

### Question Object Requirements

Each question must use **canonical keys** only—legacy keys are strictly prohibited. The required structure includes:

| Key | Type | Description |
|-----|------|-------------|
| `stage` | string | One of `pre`, `check`, or `post` |
| `question` | string | The prompt text shown to learners |
| `options` | array[string] | Between 2 and 6 answer choices (inclusive) |
| `correct` | integer | Zero-based index pointing to the correct option |
| `explanation` | string | Rationale explaining why the answer is correct |

### Stage Distribution Rules

The editorial process requires a specific distribution across the six questions to ensure proper assessment coverage:

- **1 `pre` question**: Assesses prior knowledge before lesson content
- **3 `check` questions**: Tests comprehension during the lesson
- **2 `post` questions**: Evaluates retention after content completion

## Validation Process in audit_lessons.py

The automated validation logic is implemented in **scripts/audit_lessons.py** (lines 29-95), which runs on every pull request to enforce schema compliance and block invalid submissions.

### JSON Parsing and Structure Checks

The validator first attempts to parse the file using `json.loads`. If parsing fails, it raises **L006**. The script accepts either a direct array of questions or an object containing a `questions` array—any other structure triggers **L006**.

### Legacy Schema Detection

To prevent deprecated formats from entering the codebase, the script explicitly checks for legacy keys (`q`, `choices`, `answer`). Presence of any legacy key immediately raises **L007**, rejecting the file and prompting migration to canonical keys (`question`, `options`, `correct`).

### Field Validation and Error Codes

For each question element, the validator performs granular checks with specific error codes:

- **L006**: Raised when canonical keys are missing or the JSON structure is invalid
- **L008**: Triggered when `options` is not an array of 2-6 strings
- **L009**: Raised when the `correct` integer points to an invalid index outside the `options` array range

## Practical Examples

### Minimal Valid quiz.json

The following demonstrates a compliant six-question structure adhering to all schema requirements including proper stage distribution:

```json
{
  "lesson": "02-linear-regression",
  "title": "Linear Regression Basics",
  "questions": [
    {
      "stage": "pre",
      "question": "What is the purpose of a loss function?",
      "options": ["To evaluate model size", "To measure prediction error", "To generate random data", "To encrypt outputs"],
      "correct": 1,
      "explanation": "A loss function quantifies how far predictions are from true values."
    },
    {
      "stage": "check",
      "question": "Which method solves the normal equation directly?",
      "options": ["Gradient descent", "Stochastic gradient descent", "Closed‑form solution", "Monte‑Carlo sampling"],
      "correct": 2,
      "explanation": "The closed‑form solution computes the exact weights without iteration."
    },
    {
      "stage": "check",
      "question": "What does over‑fitting indicate?",
      "options": ["Model under‑performance on training data", "Excellent generalisation", "High variance", "Low bias"],
      "correct": 2,
      "explanation": "Over‑fitting shows the model captures noise, leading to high variance."
    },
    {
      "stage": "check",
      "question": "Which metric is appropriate for regression?",
      "options": ["Accuracy", "Precision", "Mean Squared Error", "F1‑Score"],
      "correct": 2,
      "explanation": "MSE measures average squared error across continuous predictions."
    },
    {
      "stage": "post",
      "question": "When should you add regularisation?",
      "options": ["When training loss is low", "When validation loss > training loss", "When dataset is tiny", "Never"],
      "correct": 1,
      "explanation": "A higher validation loss suggests over‑fitting, motivating regularisation."
    },
    {
      "stage": "post",
      "question": "What does the R² score close to 1 mean?",
      "options": ["Poor fit", "Perfect fit", "Random predictions", "Under‑fitting"],
      "correct": 1,
      "explanation": "R² near 1 indicates the model explains most variance in the data."
    }
  ]
}

```

### Common Validation Errors

Using legacy keys like `q` instead of `question` triggers immediate rejection with error code L007:

```json
{
  "lesson": "03-legacy-example",
  "title": "Legacy Schema Demo",
  "questions": [
    {
      "stage": "pre",
      "q": "What is a typo?",
      "choices": ["A grammar mistake", "A missing semicolon"],
      "answer": 0,
      "explanation": "Legacy keys will be rejected."
    }
  ]
}

```

Running `python scripts/audit_lessons.py` on this file produces:

```text
L007: question[0] uses legacy schema keys ['answer', 'choices', 'q'] (canonical: ['correct', 'explanation', 'options', 'question', 'stage'])

```

## Summary

- The **quiz.json schema** requires exactly six questions with canonical keys (`stage`, `question`, `options`, `correct`, `explanation`) formally defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md)
- **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** enforces validation through specific error codes: **L006** (structure/keys), **L007** (legacy keys), **L008** (options count), and **L009** (index bounds)
- Legacy keys including `q`, `choices`, and `answer` are strictly prohibited and raise L007 errors during CI
- Each quiz must follow the stage distribution: 1 pre, 3 check, and 2 post questions to meet editorial standards
- The validation process runs automatically on every PR, blocking merges that contain invalid quiz structures

## Frequently Asked Questions

### What happens if my quiz.json uses legacy keys like "q" or "choices"?

The validator in [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) detects legacy keys during the CI pipeline and raises error code **L007**. This error identifies the specific legacy keys found and lists the required canonical alternatives, forcing contributors to update their JSON structure before the PR can merge.

### How many questions must each quiz contain?

While the validation script only guarantees that the `questions` array is non-empty and well-formed, the editorial standard defined in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) requires exactly six questions per lesson. This specific count ensures consistent pacing and assessment coverage across all 435 lessons in the curriculum.

### What are the valid values for the stage field?

The `stage` field accepts exactly three string values: `pre` for pre-assessment questions, `check` for comprehension checks during the lesson, and `post` for retention testing after content completion. The standard distribution requires one pre question, three check questions, and two post questions per quiz.

### Where is the quiz.json schema formally defined?

The authoritative schema definition lives in **AGENTS.md** at lines 88-102, which documents the exact structure, required keys, and data types according to the source code. The enforcement mechanism resides in **scripts/audit_lessons.py** (lines 29-95), which implements the linting logic that validates files against this specification during continuous integration.