How `audit_lessons.py` Validates Lesson Invariants in AI Engineering From Scratch

The audit_lessons.py script acts as a deterministic CI gate that traverses every lesson directory under phases/ and enforces structural contracts ranging from directory naming conventions to quiz schema integrity.

The AI Engineering From Scratch curriculum relies on scripts/audit_lessons.py to maintain homogeneous lesson structure across the repository. This standalone Python validator implements a fail-fast invariant checking system that eliminates manual review overhead by automatically verifying that every lesson conforms to mandatory naming, documentation, and content standards before any pull request can merge.

Four-Layer Validation Architecture

The validation logic is organized into four discrete layers that work together to provide a reproducible audit pipeline.

Discovery Layer

The iter_lesson_dirs function enumerates all lesson directories within the phases/ hierarchy. It accepts an optional --phase filter to restrict validation to a specific curriculum phase. The function applies two strict regular expressions—PHASE_DIR_RE and LESSON_DIR_RE—to ensure every directory follows the mandatory NN-slug naming pattern (for example, 01-intro and lesson-01-python-basics). Any directory that fails this regex match is excluded from further processing, preventing malformed paths from entering the validation pipeline.

Validation Layer

At the core of the system, the audit_lesson function orchestrates a cascade of specialized checkers. Each validator reports failures through the Audit.add method, which records the rule identifier (e.g., L001), lesson path, offending file, and a human-readable message.

Directory Naming (check_lesson_dir_pattern)
Validates that the lesson folder name strictly adheres to the repository's naming convention (rule L001). This ensures consistent URL slugs and sorting behavior across the curriculum.

Documentation Presence (check_docs_en_md)
Confirms that docs/en.md exists and meets four specific criteria (rules L002–L004): the file must be valid UTF-8, exceed a minimum byte size threshold defined by MIN_DOC_BYTES, and contain a top-level markdown heading (# ). This guarantees that every lesson provides readable, properly encoded documentation with clear titling.

Source Code Presence (check_code_main)
Scans the code/ subdirectory for any non-ignored file. An empty code directory triggers rule L005, enforcing the curriculum requirement that every lesson includes runnable source material.

Quiz Integrity (check_quiz)
Parses quiz.json and validates its schema against canonical keys while rejecting legacy keys. It enforces option count boundaries (MIN_OPTIONS to MAX_OPTIONS) and verifies that the correct answer index falls within valid bounds (rules L008–L009). This prevents malformed quizzes from reaching learners.

Internal Link Resolution (check_internal_links)
Uses MD_LINK_RE to extract all markdown links from en.md, then verifies that each relative path resolves to an existing file or directory (rule L010). This eliminates broken cross-references within the documentation.

Reporting Layer

After all lessons are processed, the render_report function aggregates issues by rule identifier. It prints a concise human-readable summary to stdout and, when invoked with the --json flag, emits a machine-readable JSON payload suitable for downstream CI tooling or automated GitHub issue creation.

CLI Layer

The main function wires command-line arguments (--phase, --json, --strict) and drives the discovery-validation loop. It exits with status 0 when all lesson invariants pass, or status 1 when any violation is detected, enabling standard CI pipeline integration.

Running the Lesson Validator

Execute a full audit across the entire curriculum:

python scripts/audit_lessons.py

Restrict validation to phase 03 and output structured JSON for automated reporting:

python scripts/audit_lessons.py --phase 3 --json

Integrate the exit code into a CI script:

if python scripts/audit_lessons.py; then
    echo "All lessons pass validation."
else
    echo "Found invariant violations – see report above."
    exit 1
fi

Summary

• audit_lessons.py serves as the canonical gatekeeper for the AI Engineering From Scratch curriculum, ensuring every lesson under phases/ meets structural contracts.
• The Discovery Layer uses PHASE_DIR_RE and LESSON_DIR_RE to filter valid directories before validation begins.
• The Validation Layer enforces rules L001 through L010, covering directory naming, UTF-8 documentation with H1 headings, non-empty code/ directories, valid quiz.json schemas, and resolvable internal markdown links.
• The Reporting Layer provides both human-readable summaries and JSON output via --json.
• The CLI Layer returns exit code 0 for clean runs and 1 for violations, supporting strict CI integration.

Frequently Asked Questions

What is the difference between rule L001 and rules L002–L004?

Rule L001 is enforced by check_lesson_dir_pattern and validates the lesson directory name itself (ensuring it matches the NN-slug pattern). Rules L002–L004 are enforced by check_docs_en_md and govern the contents of docs/en.md, specifically checking for file existence, UTF-8 encoding, minimum byte size (MIN_DOC_BYTES), and the presence of a top-level heading.

How does audit_lessons.py validate quiz integrity?

The check_quiz function parses quiz.json and validates it against a canonical schema, rejecting legacy keys. It verifies that the number of options per question falls between MIN_OPTIONS and MAX_OPTIONS, and that the correct answer index points to a valid option (rules L008–L009). This ensures that every quiz is machine-parseable and contains valid answer keys.

Can I validate lessons for only one phase of the curriculum?

Yes. Pass the --phase argument followed by the phase number. For example, python scripts/audit_lessons.py --phase 3 will restrict the audit to directories matching phases/03-*/, skipping all other phases. This is useful for rapid iteration during phase-specific development.

What does the --strict flag control in the CLI?

The --strict flag, handled by the main function, enables rigorous validation modes where warnings may be treated as fatal errors or additional schema constraints are applied. When combined with the exit code behavior (status 1 on any violation), --strict ensures that CI pipelines fail immediately if any lesson invariant is breached, preventing non-compliant content from merging into the main branch.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}