# How the CI/CD Pipeline Validates Lessons Using audit_lessons.py

> Learn how the CI/CD pipeline uses audit_lessons.py to validate lessons. Enforce schema compliance and prevent errors on every push and pull request for robust AI engineering.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-24

---

**The CI/CD pipeline runs a dedicated "audit" job on every push and pull request that executes [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) to enforce structural and content invariants, failing the build if any lesson violates the required schema.**

The `ai-engineering-from-scratch` repository maintains curriculum integrity through automated validation. Every change to lesson content triggers a comprehensive audit that checks directory structure, documentation, source code, and quiz schemas before code can be merged.

## CI Workflow Integration

The audit step is defined in **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)**. It runs on `ubuntu-latest` using Python 3.12 and triggers on both `push` and `pull_request` events for any changes under `phases/**` or the script itself.

```yaml
jobs:
  audit:
    name: invariant checks
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@...
      - uses: actions/setup-python@...
        with:
          python-version: "3.12"
      - name: run scripts/audit_lessons.py
        run: python3 scripts/audit_lessons.py

```

If [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) exits with a non-zero status, the job fails and blocks the merge.

## What audit_lessons.py Checks

The script walks the `phases/` tree (defined as `ROOT / "phases"`), optionally filtering by a single phase number with `--phase`. For each lesson directory, it performs the following invariant checks:

### Directory Naming Conventions

The function **`check_lesson_dir_pattern`** (lines 85‑94) validates that every lesson directory matches the pattern `NN-slug` (two-digit phase number, hyphen, and URL-friendly slug).

### Documentation Standards

The function **`check_docs_en_md`** (lines 97‑116) enforces that [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) exists, is valid UTF-8, contains at least 200 bytes, and includes a level-1 heading (`# …`).

### Source Code Presence

The function **`check_code_main`** (lines 119‑126) verifies the `code/` subdirectory contains at least one non-ignored file (such as [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py)), preventing empty lesson scaffolds from being merged.

### Quiz Schema Validation

The function **`check_quiz`** (lines 129‑194) validates [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) against a canonical schema. The JSON must include `stage`, `question`, `options` (containing 2‑6 items), `correct` (a valid index into the options array), and `explanation` fields.

### Internal Link Integrity

The function **`check_internal_links`** (lines 196‑211) parses [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) for relative markdown links and verifies each resolves to an existing file within the repository.

### Reporting and Exit Codes

The **`render_report`** and **`main`** functions (lines 25‑73) accumulate violations as `Issue` objects with rule identifiers like `L001` or `L002`. The script prints a human-readable summary:

```text
audit_lessons.py — 123 lesson(s) checked, 4 issue(s)

  [L002] phases/03-nlp/01-tokenizer/docs/en.md: missing docs/en.md
  [L008] phases/07-attention/02-multihead/quiz.json: options length must be 2..6 ...

Summary by rule:
  L002: 1
  L008: 2
  L010: 1

```

When invoked without flags, the script defaults to human-readable output and returns exit code **1** if any issues exist, automatically marking the CI job as failed.

## Running the Audit Locally

You can execute the validation pipeline on your development machine to catch issues before pushing.

Validate the entire curriculum:

```bash
python3 scripts/audit_lessons.py

```

Validate only a specific phase:

```bash
python3 scripts/audit_lessons.py --phase 12

```

Get machine-readable JSON output for integration with other tools:

```bash
python3 scripts/audit_lessons.py --json

```

The command returns exit status **0** when all lessons pass, otherwise **1**.

## Summary

- The audit runs automatically via GitHub Actions on every PR targeting `phases/**` or the script itself
- Six invariant checks enforce directory naming, documentation standards, code presence, quiz validity, and link integrity
- Non-zero exit codes block merges, preventing broken content from reaching learners
- Local execution supports `--phase` filtering and `--json` output for debugging

## Frequently Asked Questions

### What happens if audit_lessons.py finds an issue during CI?

The script exits with status 1, causing the GitHub Actions job to fail and block the pull request merge until the violation is fixed.

### Can I run the audit on just one phase instead of the entire curriculum?

Yes, use the `--phase` flag followed by the phase number, for example: `python3 scripts/audit_lessons.py --phase 12`.

### What Python version does the CI use to run the audit?

The workflow specifies Python 3.12 in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml) via the `actions/setup-python` action.

### Does the audit check external URLs in lesson documentation?

No, `check_internal_links` only validates relative links to files within the repository. External URLs are not verified by this script.