How the AI Engineering Quiz System Uses Pre/Check/Post Stages to Validate Learning

The rohitg00/ai-engineering-from-scratch curriculum implements a three-stage JSON-based quiz system that validates learner readiness before lessons, confirms comprehension during instruction, and verifies retention after practical application.

The repository uses a structured assessment workflow to ensure learners progress through material with validated understanding at each critical juncture. This pre/check/post stage architecture aligns with the curriculum's "pre → build → use → ship" teaching philosophy. Each stage serves a distinct pedagogical purpose enforced by automated audit scripts that validate quiz structure against a canonical schema.

The Three-Stage Validation Model

The quiz system evaluates learners at three specific points in every lesson lifecycle:

Pre-Stage: Prerequisite Assessment

The pre stage runs before learners consume lesson content. It verifies that students possess the necessary foundational knowledge or correct expectations for the upcoming material. This diagnostic check prevents learners from advancing into complex topics without adequate preparation.

Check-Stage: Comprehension Validation

The check stage executes immediately after the core "Build It" portion of a lesson. It validates that learners have understood new concepts and can apply them correctly before proceeding to practical exercises. This immediate feedback loop catches misconceptions early in the learning process.

Post-Stage: Retention Verification

The post stage occurs at the lesson's conclusion, following the "Use It" section. It reinforces long-term retention and ensures learners can recall skills after practical application. This final checkpoint confirms that knowledge has transferred from short-term working memory to durable understanding.

Canonical Quiz Schema and Audit Logic

All quizzes conform to a strict schema defined in scripts/audit_lessons.py. The script enforces that every quiz.json contains an array of question objects with exact keys:

CANONICAL_QUIZ_KEYS = {"stage", "question", "options", "correct", "explanation"}

This validation occurs at line 30 of scripts/audit_lessons.py. The audit system also verifies that each question contains 2-6 options and that the correct index points to a valid option (lines 76-89). Each question must include a "stage" property with a value of either "pre", "check", or "post", as demonstrated in lessons like phases/19-capstone-projects/84-refusal-evaluation/quiz.json.

Minimal Quiz Structure

A valid quiz.json file contains questions for all three stages:

{
  "questions": [
    {
      "stage": "pre",
      "question": "What is the purpose of a tokenizer?",
      "options": [
        "Split text into words",
        "Map tokens to ids",
        "Compress data",
        "Encrypt text"
      ],
      "correct": 1,
      "explanation": "A tokenizer maps text tokens to integer IDs for model input."
    },
    {
      "stage": "check",
      "question": "Which data structure is used for a transformer’s KV cache?",
      "options": ["List", "Dictionary", "Tensor", "Set"],
      "correct": 2,
      "explanation": "A KV cache is a tensor storing keys and values across layers."
    },
    {
      "stage": "post",
      "question": "True or false: BPE tokenizers are deterministic.",
      "options": ["True", "False"],
      "correct": 0,
      "explanation": "BPE tokenization is deterministic given a fixed merge table."
    }
  ]
}

Aggregating Quiz Data for the Learning Platform

The build script scripts/build_catalog.py collects all quiz.json files when generating the site. It groups questions by stage so that the UI can display pre-checks, in-lesson checks, and post-lesson reviews separately. This aggregation ensures consistent presentation across all lessons while maintaining the logical separation of assessment timing.

Runtime Implementation and Frontend Rendering

The frontend generator site/build.js renders each stage as a distinct quiz block. When learners submit answers, client-side JavaScript compares the submitted index against the stored correct value, displays the explanation, and records pass/fail metrics for analytics. This runtime validation happens entirely in the browser using the JSON data structure, requiring no backend server for quiz logic.

Programmatically Filtering Quizzes by Stage

You can load and filter quiz data using standard Python libraries:

import json
from pathlib import Path
from typing import List, Dict

def load_quiz(quiz_path: Path) -> List[Dict]:
    data = json.loads(quiz_path.read_text())
    return data["questions"]

def questions_by_stage(questions: List[Dict], stage: str) -> List[Dict]:
    return [q for q in questions if q["stage"] == stage]

# Example usage

quiz_file = Path("phases/19-capstone-projects/84-refusal-evaluation/quiz.json")
qs = load_quiz(quiz_file)

print("Pre‑lesson questions:", questions_by_stage(qs, "pre"))
print("In‑lesson checks:",   questions_by_stage(qs, "check"))
print("Post‑lesson recap:",   questions_by_stage(qs, "post"))

This extensible architecture allows new lesson authors to add questions by simply including the appropriate "stage" value; the audit script automatically flags missing fields or legacy keys like "q", "choices", or "answer".

Summary

• Pre-stage quizzes validate prerequisite knowledge before learners begin new material.
• Check-stage quizzes confirm immediate comprehension after the "Build It" instruction phase.
• Post-stage quizzes verify long-term retention following the "Use It" practical application.
• Schema validation in scripts/audit_lessons.py enforces consistent JSON structure with the CANONICAL_QUIZ_KEYS set.
• Build aggregation via scripts/build_catalog.py groups questions by stage for frontend rendering.
• Frontend implementation in site/build.js handles answer validation and analytics recording client-side.

Frequently Asked Questions

What distinguishes the pre-stage from the check-stage quiz?

The pre-stage assesses incoming knowledge before content delivery, while the check-stage validates understanding immediately after core instruction. The pre-stage prevents learners from starting lessons for which they lack prerequisites, whereas the check-stage catches misconceptions before learners proceed to practical exercises.

How does the audit script validate quiz JSON files?

The scripts/audit_lessons.py file enforces schema compliance by checking that every question object contains exactly the CANONICAL_QUIZ_KEYS: stage, question, options, correct, and explanation. It validates that the stage value is one of the three allowed strings and verifies that the correct index falls within the bounds of the options array (lines 76-89).

Can lesson authors create custom quiz stages beyond pre/check/post?

No, the schema restricts the "stage" field to only "pre", "check", or "post" values. The audit script will reject any quiz JSON containing alternative stage names, ensuring pedagogical consistency across the entire curriculum. Authors must map their assessment needs to these three canonical timing points.

How are quiz results tracked for learner analytics?

When learners submit answers, the frontend JavaScript compares selected indices against the correct values stored in the JSON, then records pass/fail metrics. This client-side processing enables immediate feedback while capturing performance data for progress tracking throughout the course.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}