How the quiz.json Schema Works in AI Engineering From Scratch: Stage Requirements and Validation Rules

The quiz.json file in the ai-engineering-from-scratch repository follows a strict schema requiring exactly six questions per lesson—one pre-stage, three check-stage, and two post-stage questions—each containing question, options (four choices), correct (zero-based index), and explanation fields.

The ai-engineering-from-scratch curriculum uses a standardized quiz.json schema to ensure consistent assessment across all lessons. Each quiz file must adhere to specific structural requirements and question stage distributions defined in the repository's AGENTS.md contract. Understanding this schema is essential for contributors adding new lessons or modifying existing assessments.

Understanding the quiz.json Schema Structure

Top-Level Properties

Every quiz.json file must contain three top-level keys:

• lesson: A string representing the directory slug (e.g., 13-question-answering)
• title: A human-readable string describing the lesson
• questions: An array containing exactly six question objects

Question Object Fields

Each question object within the questions array requires five specific fields:

Field	Type	Description
stage	string	One of pre, check, or post
question	string	The prompt presented to the learner
options	array	Exactly four answer choice strings
correct	integer	Zero-based index (0-3) of the correct answer
explanation	string	Rationale for the correct answer (can be empty)

Question Stage Requirements

The curriculum uses three distinct stages to assess learning progression. The stage field determines when a question appears in the learner's journey, and the counts are strictly enforced by the validator.

Pre-Stage Questions

Pre-stage questions appear before the learner engages with lesson content to gauge prior knowledge. Exactly one pre-stage question is required per quiz. These establish a baseline for measuring learning gains.

Check-Stage Questions

Check-stage questions are interspersed throughout the lesson to verify immediate comprehension of the material just presented. The schema requires exactly three check-stage questions, allowing for multiple knowledge checkpoints during the lesson flow.

Post-Stage Questions

Post-stage questions appear after the learner completes the lesson to measure retained knowledge and overall understanding. Each quiz must include exactly two post-stage questions to assess learning outcomes.

Creating a Valid quiz.json File

Below is a complete example showing how to generate a properly structured quiz.json file that satisfies all stage requirements:

import json

quiz = {
    "lesson": "99-sample-lesson",
    "title": "Sample Lesson Title",
    "questions": [
        # Exactly 1 pre-stage question

        {
            "stage": "pre",
            "question": "What do you already know about this topic?",
            "options": ["Nothing", "Basics", "Intermediate", "Expert"],
            "correct": 0,
            "explanation": "This is a diagnostic question with no wrong answer."
        },
        # Exactly 3 check-stage questions

        {
            "stage": "check",
            "question": "Which algorithm is best for this task?",
            "options": ["Linear Regression", "BERT", "K-Means", "Decision Tree"],
            "correct": 1,
            "explanation": "BERT is specifically designed for NLP tasks."
        },
        {
            "stage": "check",
            "question": "What does RAG stand for?",
            "options": ["Random Access Grade", "Retrieval-Augmented Generation", "Recursive Algorithm Graph", "None of the above"],
            "correct": 1,
            "explanation": "RAG combines retrieval systems with generative models."
        },
        {
            "stage": "check",
            "question": "Which metric evaluates QA systems?",
            "options": ["Accuracy", "BLEU", "F1 Score", "All of the above"],
            "correct": 3,
            "explanation": "Multiple metrics are used depending on the task."
        },
        # Exactly 2 post-stage questions

        {
            "stage": "post",
            "question": "When should you use fine-tuning vs RAG?",
            "options": ["Always fine-tune", "Always RAG", "Depends on data", "Neither works"],
            "correct": 2,
            "explanation": "The choice depends on your specific data and latency requirements."
        },
        {
            "stage": "post",
            "question": "What is the main bottleneck in semantic search?",
            "options": ["Embedding generation", "Vector database latency", "Query parsing", "All of the above"],
            "correct": 1,
            "explanation": "Vector database latency often dominates retrieval time."
        }
    ]
}

with open("quiz.json", "w") as f:
    json.dump(quiz, f, indent=2)

Validating quiz.json Against the Schema

The repository includes validation logic in scripts/audit_lessons.py to enforce the schema during CI checks. You can implement a standalone validator to check stage counts before committing:

import json
import collections
import pathlib

def validate_quiz(path: pathlib.Path):
    """Validate quiz.json stage distribution and structure."""
    data = json.loads(path.read_text())
    
    assert "questions" in data, "Missing questions list"
    assert len(data["questions"]) == 6, "Must have exactly 6 questions"
    
    stages = collections.Counter(q["stage"] for q in data["questions"])
    required = {"pre": 1, "check": 3, "post": 2}
    
    if stages != required:
        raise ValueError(f"Invalid stage distribution: {stages}. Required: {required}")
    
    for i, q in enumerate(data["questions"]):
        assert 0 <= q["correct"] < len(q["options"]), \
            f"Question {i}: correct index out of range"
        assert len(q["options"]) == 4, \
            f"Question {i}: must have exactly 4 options"
        assert "explanation" in q, \
            f"Question {i}: missing explanation field"
    
    print("✅ Quiz validation passed")
    return data

# Usage

if __name__ == "__main__":
    validate_quiz(pathlib.Path("quiz.json"))

Key Implementation Files

Understanding the quiz.json schema requires familiarity with these specific files in the rohitg00/ai-engineering-from-scratch repository:

• AGENTS.md: Defines the complete curriculum contract, including the exact quiz.json schema specifications and stage requirements
• scripts/audit_lessons.py: CI helper that validates every quiz.json against the schema during pull request checks
• phases/05-nlp-foundations-to-advanced/13-question-answering/quiz.json: Reference implementation showing a fully populated quiz with proper pre-, check-, and post-stage distribution

Summary

• Exactly six questions are required per lesson: one pre-stage, three check-stage, and two post-stage questions
• Four options must be provided for each question, with the correct field containing a zero-based index (0-3)
• Explanation field is required by the validator but may contain an empty string if preferred
• File location must be quiz.json in the lesson directory (e.g., phases/<phase>/<lesson>/quiz.json)
• Validation is enforced automatically by scripts/audit_lessons.py in the CI pipeline

Frequently Asked Questions

What happens if I include the wrong number of questions or stages?

The site renderer will reject the quiz, and the CI pipeline will fail. The validator in scripts/audit_lessons.py specifically checks that the stage distribution matches exactly one pre, three check, and two post questions. Any deviation triggers a ValueError during the audit process.

Can I reorder the questions in the array?

Yes, the ordering of question objects within the questions array does not affect validation. However, the stage counts must still satisfy the 1-3-2 distribution regardless of position. The curriculum platform may render questions in the order they appear, so logical grouping by stage is recommended for readability.

Is the explanation field truly optional?

While the explanation field must be present in every question object (the validator checks for its existence), the string content can be empty (""). This allows contributors to include explanatory text where helpful while maintaining schema compliance for questions where explanations are unnecessary.

How does the CI validate quiz.json files automatically?

The scripts/audit_lessons.py script traverses all lesson directories under phases/ and validates each quiz.json against the schema. It checks for JSON syntax errors, ensures exactly six questions exist, verifies the stage distribution counters, and validates that the correct index falls within the bounds of the options array. Failures block pull request merges.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}