How the CI/CD Pipeline Validates Lessons Using audit_lessons.py

The CI/CD pipeline runs a dedicated "audit" job on every push and pull request that executes scripts/audit_lessons.py to enforce structural and content invariants, failing the build if any lesson violates the required schema.

The ai-engineering-from-scratch repository maintains curriculum integrity through automated validation. Every change to lesson content triggers a comprehensive audit that checks directory structure, documentation, source code, and quiz schemas before code can be merged.

CI Workflow Integration

The audit step is defined in .github/workflows/curriculum.yml. It runs on ubuntu-latest using Python 3.12 and triggers on both push and pull_request events for any changes under phases/** or the script itself.

jobs:
  audit:
    name: invariant checks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@...
      - uses: actions/setup-python@...
        with:
          python-version: "3.12"
      - name: run scripts/audit_lessons.py
        run: python3 scripts/audit_lessons.py

If audit_lessons.py exits with a non-zero status, the job fails and blocks the merge.

What audit_lessons.py Checks

The script walks the phases/ tree (defined as ROOT / "phases"), optionally filtering by a single phase number with --phase. For each lesson directory, it performs the following invariant checks:

Directory Naming Conventions

The function check_lesson_dir_pattern (lines 85‑94) validates that every lesson directory matches the pattern NN-slug (two-digit phase number, hyphen, and URL-friendly slug).

Documentation Standards

The function check_docs_en_md (lines 97‑116) enforces that docs/en.md exists, is valid UTF-8, contains at least 200 bytes, and includes a level-1 heading (# …).

Source Code Presence

The function check_code_main (lines 119‑126) verifies the code/ subdirectory contains at least one non-ignored file (such as main.py), preventing empty lesson scaffolds from being merged.

Quiz Schema Validation

The function check_quiz (lines 129‑194) validates quiz.json against a canonical schema. The JSON must include stage, question, options (containing 2‑6 items), correct (a valid index into the options array), and explanation fields.

Internal Link Integrity

The function check_internal_links (lines 196‑211) parses docs/en.md for relative markdown links and verifies each resolves to an existing file within the repository.

Reporting and Exit Codes

The render_report and main functions (lines 25‑73) accumulate violations as Issue objects with rule identifiers like L001 or L002. The script prints a human-readable summary:

audit_lessons.py — 123 lesson(s) checked, 4 issue(s)

  [L002] phases/03-nlp/01-tokenizer/docs/en.md: missing docs/en.md
  [L008] phases/07-attention/02-multihead/quiz.json: options length must be 2..6 ...

Summary by rule:
  L002: 1
  L008: 2
  L010: 1

When invoked without flags, the script defaults to human-readable output and returns exit code 1 if any issues exist, automatically marking the CI job as failed.

Running the Audit Locally

You can execute the validation pipeline on your development machine to catch issues before pushing.

Validate the entire curriculum:

python3 scripts/audit_lessons.py

Validate only a specific phase:

python3 scripts/audit_lessons.py --phase 12

Get machine-readable JSON output for integration with other tools:

python3 scripts/audit_lessons.py --json

The command returns exit status 0 when all lessons pass, otherwise 1.

Summary

• The audit runs automatically via GitHub Actions on every PR targeting phases/** or the script itself
• Six invariant checks enforce directory naming, documentation standards, code presence, quiz validity, and link integrity
• Non-zero exit codes block merges, preventing broken content from reaching learners
• Local execution supports --phase filtering and --json output for debugging

Frequently Asked Questions

What happens if audit_lessons.py finds an issue during CI?

The script exits with status 1, causing the GitHub Actions job to fail and block the pull request merge until the violation is fixed.

Can I run the audit on just one phase instead of the entire curriculum?

Yes, use the --phase flag followed by the phase number, for example: python3 scripts/audit_lessons.py --phase 12.

What Python version does the CI use to run the audit?

The workflow specifies Python 3.12 in .github/workflows/curriculum.yml via the actions/setup-python action.

Does the audit check external URLs in lesson documentation?

No, check_internal_links only validates relative links to files within the repository. External URLs are not verified by this script.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}