# Automated Validation Scripts in AI Engineering From Scratch: A Complete Guide

> Explore automated validation scripts in AI engineering from scratch. Learn about audit_lessons.py and check_readme_counts.py for schema invariants and link integrity. Enhance your AI projects.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-07-30

---

**The `ai-engineering-from-scratch` repository provides six stand-alone automated validation scripts in the `scripts/` directory—including [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) for curriculum structure validation and [`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py) for metadata synchronization—that enforce schema invariants and link integrity using only the Python standard library.**

The `rohitg00/ai-engineering-from-scratch` curriculum relies on a robust suite of **automated validation scripts** located under the `scripts/` directory. These pure-Python utilities run locally or in CI pipelines to prevent silent drift in lesson structure, documentation counts, and cross-references. Every script operates without external dependencies, making them lightweight tools for maintaining consistency across the entire codebase.

## Core Curriculum Validators

### Validate Lesson Structure with [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py)

The [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) script serves as the primary gatekeeper for lesson folder integrity. It verifies that each lesson follows the `NN-slug` naming convention, contains a minimum-size [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file, and includes at least one source file in the `code/` directory. The script also validates [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) schema integrity, checking for correct option counts and valid `correct` indices, while ensuring all internal markdown links resolve to existing files.

Run the validator to produce a human-readable report:

```bash
python scripts/audit_lessons.py

```

For machine-consumable output in CI environments:

```bash
python scripts/audit_lessons.py --json

```

### Synchronize README Statistics with [`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py)

The [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) utility ensures hard-coded statistics in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) remain synchronized with the generated [`catalog.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/catalog.json). It parses regular expression patterns to locate badge URLs, alt-text, and prose numbers, comparing each captured value against `catalog.json.totals`. Additionally, it verifies the three "book volume" tables across [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), [`book/README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/README.md), and [`site/index.html`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/index.html) for consistency.

Check for drift with plain text output that exits with code 1 on mismatch:

```bash
python scripts/check_readme_counts.py

```

Generate JSON reports for automation:

```bash
python scripts/check_readme_counts.py --json

```

Automatically fix discrepancies locally (never use in CI):

```bash
python scripts/check_readme_counts.py --fix

```

## Extended Automation Utilities

### Verify Link Integrity with [`link_check.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/link_check.py)

The [`scripts/link_check.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/link_check.py) script performs sanity checks on external and intra-repository links within markdown files. It scans for `[...](URL)` constructs, performs HTTP HEAD requests for external URLs, and resolves relative paths against the repository root. This prevents broken references from reaching production.

Validate a specific markdown file:

```bash
python scripts/link_check.py docs/intro.md

```

### Execute Lesson Demos with [`lesson_run.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson_run.py)

For local testing, [`scripts/lesson_run.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/lesson_run.py) detects and executes a lesson's canonical demo and unit tests. According to the source code, it locates the `code/main.*` entry point, runs the implementation (e.g., `python3 main.py`), and invokes the appropriate test runner (e.g., `python -m unittest discover`).

Run a specific lesson's demo and tests:

```bash
python scripts/lesson_run.py phases/11-llm-engineering/09-function-calling

```

### Manage Skill Artifacts with [`install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/install_skills.py)

The [`scripts/install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/install_skills.py) script handles cross-lesson dependency management by populating the `outputs/` folder with compiled skill artifacts. It reads a lesson's `outputs/skill-*.md` markdown description and copies the file to the global `skills/` cache, optionally converting it to JSON format for downstream consumption.

### Bootstrap Development Environments with [`scaffold_workbench.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scaffold_workbench.py)

When creating new lessons, [`scripts/scaffold_workbench.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/scaffold_workbench.py) generates minimal development environments. It parses the lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) front-matter to infer required languages and creates [`requirements.txt`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/requirements.txt) (Python) or [`package.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/package.json) (TypeScript) files respecting the repository's dependency allowlist.

Scaffold a new lesson environment:

```bash
python scripts/scaffold_workbench.py phases/20-new-phase/01-new-lesson

```

## CI/CD Pipeline Integration

These automated validation scripts integrate directly into continuous integration workflows. As implemented in `rohitg00/ai-engineering-from-scratch`, [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) runs on every pull request and fails the **`audit`** job if any structural issue is detected. The **`readme-counts-sync`** job executes [`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py) without the `--fix` flag, ensuring any metadata drift causes an immediate build failure. This architecture prevents inconsistent curriculum changes from merging into the main branch.

## Supporting Build Scripts

Several auxiliary scripts generate the data sources required by validators:

- **[`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py)**: Generates [`catalog.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/catalog.json), the source of truth for counts used by [`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py)
- **[`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py)**: Produces `book/` artifacts including [`volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/volumes.json), which is also validated by [`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py)

## Summary

- The `scripts/` directory contains **six primary automated validation scripts** that enforce curriculum integrity without external dependencies.
- **[`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py)** validates lesson structure, quiz schemas, and internal links using the `NN-slug` folder convention.
- **[`check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/check_readme_counts.py)** synchronizes hard-coded README statistics with [`catalog.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/catalog.json) and detects metadata drift across documentation files.
- **[`link_check.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/link_check.py)**, **[`lesson_run.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lesson_run.py)**, **[`install_skills.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/install_skills.py)**, and **[`scaffold_workbench.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scaffold_workbench.py)** provide extended functionality for link verification, lesson execution, artifact management, and environment bootstrapping.
- All scripts return non-zero exit codes on failure, making them ideal for CI/CD integration.

## Frequently Asked Questions

### What does [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) validate besides folder naming?

According to the source code in [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py), the script verifies the presence and minimum size of [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), ensures at least one source file exists in `code/`, validates [`quiz.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/quiz.json) schema including correct option counts and valid `correct` indices, and confirms all internal markdown links resolve to existing files.

### Can I automatically fix README count mismatches?

Yes, by running `python scripts/check_readme_counts.py --fix` locally. However, the CI pipeline explicitly runs the script without this flag to prevent silent corrections, ensuring human review of any statistics drift before merging.

### How do I validate a single lesson's code before submitting a PR?

Use [`scripts/lesson_run.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/lesson_run.py) with the lesson path as an argument. For example: `python scripts/lesson_run.py phases/11-llm-engineering/09-function-calling`. This executes the lesson's `code/main.*` entry point and runs the unit test suite using the appropriate language-specific runner.

### Are these validation scripts dependent on external libraries?

No. All automated validation scripts in the `ai-engineering-from-scratch` repository are designed as pure-Python utilities relying solely on the standard library. This design choice ensures they run consistently across different environments without requiring dependency installation.