# How the curriculum.yml Workflow Orchestrates CI Jobs in ai-engineering-from-scratch

> Learn how curriculum.yml orchestrates CI jobs in ai-engineering-from-scratch. Discover automated validation, documentation sync, and index regeneration for a self-healing pipeline.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-14

---

**The curriculum.yml workflow orchestrates a self-healing CI pipeline through four specialized jobs that validate lesson integrity, synchronize documentation, and regenerate site indexes automatically upon code changes.**

The [`curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/curriculum.yml) workflow serves as the central CI orchestrator for the `ai-engineering-from-scratch` repository. Defined in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml), this workflow coordinates how the curriculum.yml workflow orchestrates CI jobs across the repository. It triggers on specific GitHub events to ensure that lesson content remains synchronized with generated artifacts.

## Workflow Triggers and Event Filtering

The workflow activates on two distinct event types, each constrained by path filtering to prevent unnecessary runs. According to the source code in [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml), pushes to the `main` branch trigger execution when modifying lesson files, scripts, documentation, or the workflow itself (lines 4-15). Similarly, pull requests targeting `main` initiate checks when the same file paths change (lines 16-27).

This path-specific filtering ensures that unrelated changes—such as updates to CI configuration or external documentation—do not consume runner resources. The **event filtering** logic guarantees that the heavy CI jobs only execute when curriculum content actually changes.

## The Four-Job CI Matrix

The workflow defines four specialized jobs running on `ubuntu-latest` runners, each designed for a specific validation or synchronization task.

### Audit Job: Invariant Checking

The **audit** job executes repository-wide invariant checks via [`scripts/audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/audit_lessons.py). This job runs on every push event (lines 32-34) and performs the following steps:

- Checks out the repository code (lines 37-40)
- Sets up Python 3.12 (lines 40-42)
- Executes the audit script to validate lesson structure and dependencies (lines 43-44)

This job acts as the first line of defense, ensuring that all curriculum files adhere to structural requirements before any automated fixes are applied.

### readme-counts-sync: Automated Documentation Repair

The **readme-counts-sync** job automatically repairs lesson count tables in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). Triggered exclusively on push events (lines 46-48), this job uses [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) with the `--fix` flag to correct discrepancies. The process includes:

1. Checking out the code and setting up Python 3.12 (lines 53-60)
2. Building the internal catalog using [`scripts/build_catalog.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_catalog.py) (lines 60-61)
3. Running the check script with automatic fixing enabled (lines 62-63)
4. Committing and pushing changes back to the repository if modifications are detected (lines 65-98)

This job ensures that curriculum statistics displayed in the README remain accurate without manual intervention.

### site-rebuild: Site Index Generation

The **site-rebuild** job regenerates [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), the data source powering the curriculum website. This job depends on the successful completion of `readme-counts-sync` (line 105) and only runs on push events (lines 100-106). The execution flow includes:

- Checking out the repository (lines 108-111)
- Executing `node site/build.js` to generate the site index (lines 112-113)
- Committing and pushing the updated [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) if changes exist (lines 115-147)

By declaring `needs: readme-counts-sync`, the workflow guarantees that the README is already up-to-date before the site data is regenerated.

### readme-counts-drift: Pull Request Validation

For pull requests, the workflow executes the **readme-counts-drift** job instead of the sync jobs. This lightweight check (lines 149-152) verifies whether [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) counts are out-of-sync with the current lesson state without modifying the repository. The process runs [`scripts/check_readme_counts.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/check_readme_counts.py) without the `--fix` flag, reporting warnings if drift is detected while keeping the review process non-intrusive.

## Orchestration Mechanisms and Safety Features

The [`curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/curriculum.yml) workflow implements several sophisticated mechanisms to prevent conflicts and ensure reliable CI job orchestration.

**Job Dependencies** create a linear execution chain where `site-rebuild` waits for `readme-counts-sync` to complete (line 105). This sequencing ensures data consistency across generated artifacts.

**Permission Scoping** restricts write access to only the jobs that require it. The workflow requests `contents: read` globally (lines 29-31) while escalating to `contents: write` specifically for jobs that commit changes back to the repository.

**Idempotent Bot Commits** prevent infinite loops by checking the latest commit message for bot prefixes. Both the README sync and site rebuild jobs abort if the previous commit was already bot-generated (lines 73-76 and 123-125), avoiding circular CI triggers.

**Retry Logic** handles race conditions on the `main` branch through exponential backoff. When pushing changes, the jobs attempt up to five retries (lines 83-96 and 132-145), ensuring that temporary conflicts do not fail the entire pipeline.

## Running the CI Pipeline Locally and Manually

Triggering the full CI pipeline requires pushing curriculum changes that match the path filters:

```bash
git add phases/01-foundations/lesson-01/
git commit -m "feat(phase-01/01): add new lesson"
git push origin main

```

This push matches the `phases/**` path filter and automatically runs the **audit**, **readme-counts-sync**, and **site-rebuild** jobs.

To check CI results for a pull request locally:

```bash
gh pr checkout 42
git push origin HEAD:refs/heads/feature-branch

```

When the PR opens, GitHub executes the **readme-counts-drift** job and surfaces warnings if the README requires updates.

Because the jobs are idempotent, manually re-running them through the GitHub UI will not create duplicate commits if the fixes are already applied:

```yaml

# In the GitHub UI, go to Actions → curriculum → Re-run jobs

```

## Summary

- The [`.github/workflows/curriculum.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/curriculum.yml) file defines a four-job CI matrix that validates and repairs curriculum documentation automatically.
- **Event filtering** ensures jobs only run when lesson files, scripts, or documentation change, conserving runner resources.
- The **audit** job validates structural integrity, while **readme-counts-sync** and **site-rebuild** maintain automated documentation and site indexes on pushes.
- **readme-counts-drift** provides lightweight validation for pull requests without modifying the repository.
- **Safety mechanisms** including job dependencies, idempotent bot commits, and retry logic with exponential backoff prevent race conditions and infinite loops.

## Frequently Asked Questions

### Why does the site-rebuild job depend on readme-counts-sync?

The `site-rebuild` job declares `needs: readme-counts-sync` (line 105) to ensure that [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) lesson counts are accurate before regenerating the site index. This dependency guarantees that the [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) file reflects the most current curriculum state, preventing synchronization errors between the documentation and the website.

### How does the workflow prevent infinite loops when auto-committing changes?

Both the `readme-counts-sync` and `site-rebuild` jobs implement idempotency checks by examining the latest commit message for bot prefixes. If the previous commit was already generated by automation (lines 73-76 and 123-125), the jobs abort to avoid creating circular triggers that would endlessly restart the CI pipeline.

### What is the difference between readme-counts-sync and readme-counts-drift?

The `readme-counts-sync` job runs on push events and automatically fixes [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) lesson counts using the `--fix` flag, committing changes back to the repository. In contrast, `readme-counts-drift` runs exclusively on pull requests and performs a read-only check to warn about synchronization issues without modifying files, keeping the review process fast and non-intrusive.

### What triggers the curriculum.yml workflow in ai-engineering-from-scratch?

The workflow triggers on two events: pushes to the `main` branch and pull requests targeting `main`, but only when these events modify specific paths including lesson files, scripts, or documentation. This path filtering prevents the CI jobs from running on unrelated changes like configuration updates or external assets.