# How `audit_lessons.py` Validates Lesson Invariants in AI Engineering From Scratch

> Learn how audit_lessons.py validates lesson invariants in AI Engineering From Scratch. This script ensures structural contracts and quiz schema integrity for robust AI development.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-09-01

---

**The [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) script acts as a deterministic CI gate that traverses every lesson directory under `phases/` and enforces structural contracts ranging from directory naming conventions to quiz schema integrity.**

The *AI Engineering From Scratch* curriculum relies on [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) to maintain homogeneous lesson structure across the repository. This standalone Python validator implements a **fail-fast invariant checking system** that eliminates manual review overhead by automatically verifying that every lesson conforms to mandatory naming, documentation, and content standards before any pull request can merge.

## Four-Layer Validation Architecture

The validation logic is organized into four discrete layers that work together to provide a reproducible audit pipeline.

### Discovery Layer

The `iter_lesson_dirs` function enumerates all lesson directories within the `phases/` hierarchy. It accepts an optional `--phase` filter to restrict validation to a specific curriculum phase. The function applies two strict regular expressions—`PHASE_DIR_RE` and `LESSON_DIR_RE`—to ensure every directory follows the mandatory `NN-slug` naming pattern (for example, `01-intro` and `lesson-01-python-basics`). Any directory that fails this regex match is excluded from further processing, preventing malformed paths from entering the validation pipeline.

### Validation Layer

At the core of the system, the `audit_lesson` function orchestrates a cascade of specialized checkers. Each validator reports failures through the `Audit.add` method, which records the rule identifier (e.g., `L001`), lesson path, offending file, and a human-readable message.

**Directory Naming (`check_lesson_dir_pattern`)**  
Validates that the lesson folder name strictly adheres to the repository's naming convention (rule `L001`). This ensures consistent URL slugs and sorting behavior across the curriculum.

**Documentation Presence (`check_docs_en_md`)**  
Confirms that [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) exists and meets four specific criteria (rules `L002`–`L004`): the file must be valid UTF-8, exceed a minimum byte size threshold defined by `MIN_DOC_BYTES`, and contain a top-level markdown heading (`# `). This guarantees that every lesson provides readable, properly encoded documentation with clear titling.

**Source Code Presence (`check_code_main`)**  
Scans the `code/` subdirectory for any non-ignored file. An empty code directory triggers rule `L005`, enforcing the curriculum requirement that every lesson includes runnable source material.

**Quiz Integrity (`check_quiz`)**  
Parses [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) and validates its schema against canonical keys while rejecting legacy keys. It enforces option count boundaries (`MIN_OPTIONS` to `MAX_OPTIONS`) and verifies that the correct answer index falls within valid bounds (rules `L008`–`L009`). This prevents malformed quizzes from reaching learners.

**Internal Link Resolution (`check_internal_links`)**  
Uses `MD_LINK_RE` to extract all markdown links from [`en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/en.md), then verifies that each relative path resolves to an existing file or directory (rule `L010`). This eliminates broken cross-references within the documentation.

### Reporting Layer

After all lessons are processed, the `render_report` function aggregates issues by rule identifier. It prints a concise human-readable summary to stdout and, when invoked with the `--json` flag, emits a machine-readable JSON payload suitable for downstream CI tooling or automated GitHub issue creation.

### CLI Layer

The `main` function wires command-line arguments (`--phase`, `--json`, `--strict`) and drives the discovery-validation loop. It exits with status `0` when all lesson invariants pass, or status `1` when any violation is detected, enabling standard CI pipeline integration.

## Running the Lesson Validator

Execute a full audit across the entire curriculum:

```bash
python scripts/audit_lessons.py

```

Restrict validation to phase 03 and output structured JSON for automated reporting:

```bash
python scripts/audit_lessons.py --phase 3 --json

```

Integrate the exit code into a CI script:

```bash
if python scripts/audit_lessons.py; then
    echo "All lessons pass validation."
else
    echo "Found invariant violations – see report above."
    exit 1
fi

```

## Summary

- **[`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py)** serves as the canonical gatekeeper for the *AI Engineering From Scratch* curriculum, ensuring every lesson under `phases/` meets structural contracts.
- The **Discovery Layer** uses `PHASE_DIR_RE` and `LESSON_DIR_RE` to filter valid directories before validation begins.
- The **Validation Layer** enforces rules `L001` through `L010`, covering directory naming, UTF-8 documentation with H1 headings, non-empty `code/` directories, valid [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) schemas, and resolvable internal markdown links.
- The **Reporting Layer** provides both human-readable summaries and JSON output via `--json`.
- The **CLI Layer** returns exit code `0` for clean runs and `1` for violations, supporting strict CI integration.

## Frequently Asked Questions

### What is the difference between rule L001 and rules L002–L004?

**Rule L001** is enforced by `check_lesson_dir_pattern` and validates the lesson directory name itself (ensuring it matches the `NN-slug` pattern). **Rules L002–L004** are enforced by `check_docs_en_md` and govern the contents of [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), specifically checking for file existence, UTF-8 encoding, minimum byte size (`MIN_DOC_BYTES`), and the presence of a top-level heading.

### How does [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) validate quiz integrity?

The `check_quiz` function parses [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) and validates it against a canonical schema, rejecting legacy keys. It verifies that the number of options per question falls between `MIN_OPTIONS` and `MAX_OPTIONS`, and that the correct answer index points to a valid option (rules `L008`–`L009`). This ensures that every quiz is machine-parseable and contains valid answer keys.

### Can I validate lessons for only one phase of the curriculum?

Yes. Pass the `--phase` argument followed by the phase number. For example, `python scripts/audit_lessons.py --phase 3` will restrict the audit to directories matching `phases/03-*/`, skipping all other phases. This is useful for rapid iteration during phase-specific development.

### What does the `--strict` flag control in the CLI?

The `--strict` flag, handled by the `main` function, enables rigorous validation modes where warnings may be treated as fatal errors or additional schema constraints are applied. When combined with the exit code behavior (status `1` on any violation), `--strict` ensures that CI pipelines fail immediately if any lesson invariant is breached, preventing non-compliant content from merging into the main branch.