# CI/CD Workflows for AI Curriculum Validation: How the Repository Automates Content Checks and Syncs site/data.js

> Learn how the ai-engineering-from-scratch repo automates AI curriculum validation and syncs site/data.js using a single GitHub Actions workflow with four jobs.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-23

---

**The rohitg00/ai-engineering-from-scratch repository uses a single GitHub Actions workflow with four specialized jobs to automatically validate lesson integrity, regenerate documentation statistics, and rebuild the [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) index on every push to main.**

The **CI/CD workflows** in this open-source AI curriculum repository ensure that every lesson meets strict structural and content standards while keeping the public-facing site data synchronized with the underlying markdown content. Located in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml), the pipeline runs on every pull request and push to the `main` branch, providing continuous validation and automatic correction of curriculum metadata.

## The Four-Job Validation Pipeline

The workflow orchestrates four distinct jobs that enforce **curriculum invariants**, auto-fix documentation counts, and regenerate the static site index.

### The Audit Job: Content Integrity Validation

The **audit** job serves as the primary gatekeeper for curriculum quality. It executes [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) to verify that every lesson directory follows naming conventions, contains required documentation, adheres to quiz schema specifications, and maintains valid internal links.

This job runs on both `push` and `pull_request` events targeting `main`, ensuring that broken links or malformed lesson structures never reach the production branch. The Python script performs linter-style checks across the entire `phases/` directory tree, validating that the AI curriculum maintains consistent formatting standards.

### The README Counts Sync Job: Auto-Healing Documentation

The **readme-counts-sync** job automatically regenerates the temporary curriculum catalog and rewrites the lesson counters in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) (e.g., updating "435 lessons" to reflect actual content). This job triggers only on pushes to `main`, not on pull requests.

The workflow executes [`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py) followed by `scripts/check_readme_counts.py --fix`. When the script detects drift between the documented count and actual lesson files, it commits the correction directly back to the branch. This self-healing mechanism ensures the repository's primary documentation always reflects the true state of the curriculum.

### The Site Rebuild Job: Syncing site/data.js

The **site-rebuild** job recreates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), the JSON-like index that powers the static site frontend. This job depends on the successful completion of the README sync job, ensuring that documentation corrections propagate before the site data regenerates.

Running `node site/build.js`, the job traverses the `phases/` directory structure and outputs a comprehensive index of all lessons, phases, and metadata. When changes are detected, the workflow automatically commits the updated [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) back to the repository, keeping the published site in perfect sync with the curriculum source files.

### The Drift Detection Job: Pull Request Safety

The **readme-counts-drift** job provides non-blocking advisory checks on pull requests. It builds a temporary catalog using [`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py) and runs [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) without the `--fix` flag, outputting warnings if the README statistics diverge from the source.

Unlike the sync job, this validation job never commits changes. It allows contributors to see potential count mismatches before merging, maintaining transparency about documentation state without automatically modifying contributor branches.

## Running Validation Locally

Contributors can execute the same validation scripts used in the CI/CD workflows to verify changes before pushing. The repository maintains identical tooling for local development and continuous integration.

Run the full validation suite locally:

```bash

# Validate lesson structure, links, and schema

python3 scripts/audit_lessons.py

# Check if README counts match actual lessons

python3 scripts/build_catalog.py
python3 scripts/check_readme_counts.py

# Rebuild the site data index

node site/build.js

```

Trigger the CI/CD workflows manually by pushing curriculum changes:

```bash

# Stage a lesson update

git add phases/01-intro-to-ml/01-ml-overview/docs/en.md
git commit -m "fix: update lesson heading"
git push origin main

# Create a pull request to run the drift check

gh pr create --title "fix: typo in lesson 01-ml-overview" \
             --body "Runs the curriculum workflow on PR"

```

Inspect the generated site data file:

```bash
cat site/data.js | head -n 20

```

## Key Files and Their Roles

- **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)** – Central CI/CD workflow defining the audit, sync, rebuild, and drift detection jobs.
- **[`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py)** – Python validator enforcing lesson naming conventions, documentation requirements, quiz schemas, and internal link integrity.
- **[`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py)** – Generates intermediate catalog JSON used by the README sync and drift detection jobs.
- **[`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py)** – Computes expected lesson counts and optionally rewrites [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) when statistics diverge.
- **[`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)** – Node.js script that walks the `phases/` tree and writes [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), the data source for the static site frontend.
- **[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js)** – Auto-generated index of all lessons imported by the site's front-end framework.
- **[`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md)** – Documentation outlining curriculum-wide conventions and CI/CD enforcement mechanisms.

## Summary

- The repository uses **[`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml)** to orchestrate four specialized jobs on every change to `main`.
- The **audit** job runs [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py) to validate lesson structure, links, and schema on every push and pull request.
- The **readme-counts-sync** job auto-corrects lesson counters in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) by running `scripts/check_readme_counts.py --fix` on pushes only.
- The **site-rebuild** job executes `node site/build.js` to regenerate [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) automatically when the curriculum structure changes.
- The **readme-counts-drift** job provides advisory warnings on pull requests without committing changes, preventing surprise documentation updates.
- All validation scripts can be executed locally using Python 3.12 and Node.js, ensuring contributor workflows match CI/CD behavior exactly.

## Frequently Asked Questions

### What triggers the CI/CD workflows in this repository?

The **curriculum.yml** workflow triggers on every `push` to the `main` branch and every `pull_request` targeting `main`. Push events activate all four jobs including the auto-fixing sync operations, while pull requests only trigger the non-destructive audit and drift detection jobs to prevent automatic modifications to contributor branches.

### How does the site/data.js file get updated automatically?

The **site-rebuild** job depends on the README sync job and runs `node site/build.js` to traverse the `phases/` directory and generate a new JSON index. When the script detects changes to the curriculum structure, the workflow commits the updated [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) directly to the `main` branch, ensuring the static site always reflects the current lesson catalog.

### Can I run the curriculum validation locally before committing?

Yes. The repository uses the same scripts locally and in CI. Run `python3 scripts/audit_lessons.py` to check lesson integrity, `python3 scripts/build_catalog.py && python3 scripts/check_readme_counts.py` to verify documentation counts, and `node site/build.js` to preview the site data generation. These commands mirror the steps executed in the GitHub Actions workflow.

### What happens if the README lesson counts drift from the actual content?

On pushes to `main`, the **readme-counts-sync** job detects the discrepancy using `scripts/check_readme_counts.py --fix`, automatically commits the corrected count to [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md), and pushes the change. On pull requests, the **readme-counts-drift** job logs a warning without modifying files, alerting contributors to update the documentation before merging.