Automated Validation Scripts in AI Engineering From Scratch: A Complete Guide

The ai-engineering-from-scratch repository provides six stand-alone automated validation scripts in the scripts/ directory—including audit_lessons.py for curriculum structure validation and check_readme_counts.py for metadata synchronization—that enforce schema invariants and link integrity using only the Python standard library.

The rohitg00/ai-engineering-from-scratch curriculum relies on a robust suite of automated validation scripts located under the scripts/ directory. These pure-Python utilities run locally or in CI pipelines to prevent silent drift in lesson structure, documentation counts, and cross-references. Every script operates without external dependencies, making them lightweight tools for maintaining consistency across the entire codebase.

Core Curriculum Validators

Validate Lesson Structure with audit_lessons.py

The scripts/audit_lessons.py script serves as the primary gatekeeper for lesson folder integrity. It verifies that each lesson follows the NN-slug naming convention, contains a minimum-size docs/en.md file, and includes at least one source file in the code/ directory. The script also validates quiz.json schema integrity, checking for correct option counts and valid correct indices, while ensuring all internal markdown links resolve to existing files.

Run the validator to produce a human-readable report:

python scripts/audit_lessons.py

For machine-consumable output in CI environments:

python scripts/audit_lessons.py --json

Synchronize README Statistics with check_readme_counts.py

The scripts/check_readme_counts.py utility ensures hard-coded statistics in README.md remain synchronized with the generated catalog.json. It parses regular expression patterns to locate badge URLs, alt-text, and prose numbers, comparing each captured value against catalog.json.totals. Additionally, it verifies the three "book volume" tables across README.md, book/README.md, and site/index.html for consistency.

Check for drift with plain text output that exits with code 1 on mismatch:

python scripts/check_readme_counts.py

Generate JSON reports for automation:

python scripts/check_readme_counts.py --json

Automatically fix discrepancies locally (never use in CI):

python scripts/check_readme_counts.py --fix

Extended Automation Utilities

Verify Link Integrity with link_check.py

The scripts/link_check.py script performs sanity checks on external and intra-repository links within markdown files. It scans for [...](URL) constructs, performs HTTP HEAD requests for external URLs, and resolves relative paths against the repository root. This prevents broken references from reaching production.

Validate a specific markdown file:

python scripts/link_check.py docs/intro.md

Execute Lesson Demos with lesson_run.py

For local testing, scripts/lesson_run.py detects and executes a lesson's canonical demo and unit tests. According to the source code, it locates the code/main.* entry point, runs the implementation (e.g., python3 main.py), and invokes the appropriate test runner (e.g., python -m unittest discover).

Run a specific lesson's demo and tests:

python scripts/lesson_run.py phases/11-llm-engineering/09-function-calling

Manage Skill Artifacts with install_skills.py

The scripts/install_skills.py script handles cross-lesson dependency management by populating the outputs/ folder with compiled skill artifacts. It reads a lesson's outputs/skill-*.md markdown description and copies the file to the global skills/ cache, optionally converting it to JSON format for downstream consumption.

Bootstrap Development Environments with scaffold_workbench.py

When creating new lessons, scripts/scaffold_workbench.py generates minimal development environments. It parses the lesson's docs/en.md front-matter to infer required languages and creates requirements.txt (Python) or package.json (TypeScript) files respecting the repository's dependency allowlist.

Scaffold a new lesson environment:

python scripts/scaffold_workbench.py phases/20-new-phase/01-new-lesson

CI/CD Pipeline Integration

These automated validation scripts integrate directly into continuous integration workflows. As implemented in rohitg00/ai-engineering-from-scratch, audit_lessons.py runs on every pull request and fails the audit job if any structural issue is detected. The readme-counts-sync job executes check_readme_counts.py without the --fix flag, ensuring any metadata drift causes an immediate build failure. This architecture prevents inconsistent curriculum changes from merging into the main branch.

Supporting Build Scripts

Several auxiliary scripts generate the data sources required by validators:

• scripts/build_catalog.py: Generates catalog.json, the source of truth for counts used by check_readme_counts.py
• scripts/build_book.py: Produces book/ artifacts including volumes.json, which is also validated by check_readme_counts.py

Summary

• The scripts/ directory contains six primary automated validation scripts that enforce curriculum integrity without external dependencies.
• audit_lessons.py validates lesson structure, quiz schemas, and internal links using the NN-slug folder convention.
• check_readme_counts.py synchronizes hard-coded README statistics with catalog.json and detects metadata drift across documentation files.
• link_check.py, lesson_run.py, install_skills.py, and scaffold_workbench.py provide extended functionality for link verification, lesson execution, artifact management, and environment bootstrapping.
• All scripts return non-zero exit codes on failure, making them ideal for CI/CD integration.

Frequently Asked Questions

What does audit_lessons.py validate besides folder naming?

According to the source code in scripts/audit_lessons.py, the script verifies the presence and minimum size of docs/en.md, ensures at least one source file exists in code/, validates quiz.json schema including correct option counts and valid correct indices, and confirms all internal markdown links resolve to existing files.

Can I automatically fix README count mismatches?

Yes, by running python scripts/check_readme_counts.py --fix locally. However, the CI pipeline explicitly runs the script without this flag to prevent silent corrections, ensuring human review of any statistics drift before merging.

How do I validate a single lesson's code before submitting a PR?

Use scripts/lesson_run.py with the lesson path as an argument. For example: python scripts/lesson_run.py phases/11-llm-engineering/09-function-calling. This executes the lesson's code/main.* entry point and runs the unit test suite using the appropriate language-specific runner.

Are these validation scripts dependent on external libraries?

No. All automated validation scripts in the ai-engineering-from-scratch repository are designed as pure-Python utilities relying solely on the standard library. This design choice ensures they run consistently across different environments without requiring dependency installation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}