# ROADMAP.md Status Tracking vs site/data.js Auto-Generation in AI Engineering From Scratch

> Explore the differences between ROADMAP.md status tracking and site/data.js auto-generation in AI Engineering From Scratch. Understand how these methods power project progress and interactive front-ends.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-09-04

---

**The curriculum maintains [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) as the human-editable source of truth for lesson statuses using visual glyphs, while [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is a machine-generated JSON artifact created by [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) to power the interactive front-end.**

The `rohitg00/ai-engineering-from-scratch` repository employs a dual-layer architecture to separate human-friendly progress tracking from machine-readable website data. Understanding the distinction between **ROADMAP.md status tracking** and **site/data.js auto-generation** is essential for contributors who need to update lesson statuses without breaking the automated build pipeline.

## The Dual-Layer Architecture

### ROADMAP.md: Human-Readable Source of Truth

[`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) serves as the single source of truth for curriculum progress. This file contains a human-editable markdown table that lists every phase and lesson alongside status glyphs: **✅** for complete, **🚧** for in-progress, and **⬚** for planned. Contributors manually update these glyphs when lesson development status changes, making it possible to track progress at a glance without executing any build scripts.

The file maintains only lesson titles and their corresponding status emojis. It does not contain URLs, SEO metadata, or navigation structures. When you commit changes to this file (e.g., `git commit -m "feat(phase-5/01): complete tokenization lesson"`), you update the canonical record that the rest of the pipeline consumes.

### site/data.js: Machine-Generated Consumption Layer

[`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is a JSON file that the front-end reads at runtime to render the curriculum catalog, lesson pages, and discovery widgets. **Never edit this file by hand**—it is regenerated automatically on every push via the `site-rebuild` workflow defined in **AGENTS.md**. The file contains full lesson metadata including names, GitHub URLs, types, languages, status fields, and navigation links.

Any drift between the markdown roadmap and the generated data triggers a CI failure via [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py), ensuring the website cannot display stale or inconsistent information.

## How the Build Pipeline Bridges Both Layers

The transformation from markdown glyphs to structured JSON occurs in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) through a four-stage pipeline:

### Step 1: Parsing the Roadmap

The `parseRoadmap` function (line 105 in [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js)) reads [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) line-by-line using regex patterns to extract phase headers and lesson rows. It converts visual emojis into programmatic status strings:

```javascript
// site/build.js lines 105-133
function parseRoadmap(content) {
  const statuses = {};
  let currentPhase = null;
  for (const line of content.split(/\r?\n/)) {
    const phaseMatch = line.match(/^##\s+Phase\s+(\d+).*?—\s*(✅|🚧|⬚)/);
    if (phaseMatch) {
      const phaseId = parseInt(phaseMatch[1]);
      const statusEmoji = phaseMatch[2];
      currentPhase = `Phase ${phaseId}`;
      statuses[currentPhase] = {
        phaseStatus: statusEmoji === '✅' ? 'complete' :
                     statusEmoji === '🚧' ? 'in-progress' : 'planned',
        lessons: {}
      };
      continue;
    }
    if (currentPhase) {
      const lessonMatch = line.match(/^\|\s*\d+\s*\|\s*(.+?)\s*\|\s*(✅|🚧|⬚)\s*\|/);
      if (lessonMatch) {
        const lessonName = lessonMatch[1].trim();
        const statusEmoji = lessonMatch[2];
        const status = statusEmoji === '✅' ? 'complete' :
                       statusEmoji === '🚧' ? 'in-progress' : 'planned';
        statuses[currentPhase].lessons[lessonName] = status;
      }
    }
  }
  return statuses;
}

```

This function returns an object mapping phase names to their statuses and lesson collections.

### Step 2: Merging with README.md Metadata

The `parseReadme` function (line 338) processes [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) to extract canonical lesson tables containing GitHub URLs. It matches each lesson name to the roadmap entry and enforces consistency. If a lesson has a concrete GitHub link but the roadmap still shows "planned," the script forces the status to **complete** (lines 70-73):

```javascript
// site/build.js lines 70-73
if (url && status === 'planned') {
  // Any lesson that has a concrete GitHub link is at least complete.
  status = 'complete';
}

```

This validation prevents the scenario where a lesson directory exists but the roadmap incorrectly displays it as unstarted.

### Step 3: Generating the JSON Output

After assembling the complete phase and lesson objects, the script writes a compact JSON representation to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) (line 209). The `writeDataFile` function wraps the data in a global variable for browser consumption:

```javascript
// site/build.js
function writeDataFile(phases) {
  const output = `window.AIFS_DATA = ${JSON.stringify({ phases }, null, 2)};`;
  fs.writeFileSync(path.join(__dirname, 'data.js'), output, 'utf8');
}

```

The resulting file contains structured arrays with complete metadata including `id`, `name`, `status`, `type`, `lang`, and `url` for every lesson.

### Step 4: CI Enforcement and Validation

The `site-rebuild` workflow runs `node site/build.js` on every push. If the generated [`data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/data.js) does not reflect the current state of [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) (for example, if you manually edited the JSON or forgot to update the markdown glyphs), [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) fails the build. This guarantees that the website always mirrors the latest roadmap and eliminates manual copy-and-paste errors.

## Why Both Systems Exist

- **Human-friendly tracking:** Contributors can open [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) directly in the GitHub UI to see visual progress indicators without generating the site or running local scripts. The glyphs also feed the roadmap badge displayed on the public site.

- **Machine-ready consumption:** The React-based front-end requires a deterministic, serializable structure. Auto-generation ensures that URL paths, navigation links, and SEO fields remain synchronized with the actual repository structure. Without this separation, maintaining consistent URLs across 50+ lessons would require manual updates in multiple locations.

## Implementation Details

### Extracting Status Glyphs

The build script relies on specific regex patterns to identify phases and lessons. Phase headers must match the pattern `## Phase {N} ... — {emoji}`, while lesson rows follow the markdown table format `| {number} | {title} | {emoji} |`. Any deviation from this format causes the parser to skip entries, which is why the CI audit is critical.

### Validating Status Consistency

The override logic in `parseReadme` acts as a safety mechanism. When the script detects a lesson URL in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) that points to an existing directory, it automatically promotes the status from "planned" to "complete" regardless of the emoji in [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md). This ensures the website shows accurate availability even if contributors forget to update the roadmap glyphs immediately after merging a lesson.

### Writing the Generated Artifact

The output file [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) is a JavaScript module that assigns JSON to `window.AIFS_DATA`, allowing the static site to load curriculum data without additional fetch requests. This approach minimizes load times for the interactive catalog while maintaining a clear separation between source content and generated assets.

## Summary

- **ROADMAP.md** is the manual source of truth containing visual status glyphs (✅ 🚧 ⬚) for human contributors.
- **site/data.js** is the auto-generated JSON consumed by the front-end, created by [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js).
- The build pipeline parses markdown tables, validates statuses against [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) links, and writes structured data to [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js).
- CI enforcement via [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) prevents drift between the roadmap and the website.
- Never edit [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) manually—always update [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) and let the build script regenerate the JSON.

## Frequently Asked Questions

### What happens if I edit site/data.js directly?

Your changes will be overwritten on the next push. The `site-rebuild` workflow runs `node site/build.js` automatically, regenerating the file from [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) and [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). If the audit script detects manual modifications that conflict with the source markdown, the CI job fails and blocks the deployment.

### How does the build script handle status mismatches?

The `parseReadme` function contains validation logic (lines 70-73) that checks if a lesson has a GitHub URL in [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md). If a URL exists but [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) shows the lesson as "planned," the script forces the status to "complete." This ensures the website reflects actual repository contents even when the roadmap glyphs lag behind development.

### Can I add a new lesson without updating ROADMAP.md?

No. While the lesson content would exist in the repository, the website catalog and navigation structures rely on [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js), which is generated exclusively from [`ROADMAP.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/ROADMAP.md) and [`README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/README.md) parsing. Without updating the roadmap table, the build script will not include the lesson in the generated JSON, making it invisible to the site's search and navigation features.

### Where is the build automation configured?

The automation lives in two locations: the [`site/build.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) script contains the transformation logic (`parseRoadmap`, `parseReadme`, and `writeDataFile`), while the **AGENTS.md** file defines the `site-rebuild` GitHub Actions workflow that triggers the build on every push. The [`audit_lessons.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/audit_lessons.py) script provides the final validation gate to ensure data consistency.