ROADMAP.md Status Tracking vs site/data.js Auto-Generation in AI Engineering From Scratch

The curriculum maintains ROADMAP.md as the human-editable source of truth for lesson statuses using visual glyphs, while site/data.js is a machine-generated JSON artifact created by site/build.js to power the interactive front-end.

The rohitg00/ai-engineering-from-scratch repository employs a dual-layer architecture to separate human-friendly progress tracking from machine-readable website data. Understanding the distinction between ROADMAP.md status tracking and site/data.js auto-generation is essential for contributors who need to update lesson statuses without breaking the automated build pipeline.

The Dual-Layer Architecture

ROADMAP.md: Human-Readable Source of Truth

ROADMAP.md serves as the single source of truth for curriculum progress. This file contains a human-editable markdown table that lists every phase and lesson alongside status glyphs: ✅ for complete, 🚧 for in-progress, and ⬚ for planned. Contributors manually update these glyphs when lesson development status changes, making it possible to track progress at a glance without executing any build scripts.

The file maintains only lesson titles and their corresponding status emojis. It does not contain URLs, SEO metadata, or navigation structures. When you commit changes to this file (e.g., git commit -m "feat(phase-5/01): complete tokenization lesson"), you update the canonical record that the rest of the pipeline consumes.

site/data.js: Machine-Generated Consumption Layer

site/data.js is a JSON file that the front-end reads at runtime to render the curriculum catalog, lesson pages, and discovery widgets. Never edit this file by hand—it is regenerated automatically on every push via the site-rebuild workflow defined in AGENTS.md. The file contains full lesson metadata including names, GitHub URLs, types, languages, status fields, and navigation links.

Any drift between the markdown roadmap and the generated data triggers a CI failure via audit_lessons.py, ensuring the website cannot display stale or inconsistent information.

How the Build Pipeline Bridges Both Layers

The transformation from markdown glyphs to structured JSON occurs in site/build.js through a four-stage pipeline:

Step 1: Parsing the Roadmap

The parseRoadmap function (line 105 in site/build.js) reads ROADMAP.md line-by-line using regex patterns to extract phase headers and lesson rows. It converts visual emojis into programmatic status strings:

// site/build.js lines 105-133
function parseRoadmap(content) {
  const statuses = {};
  let currentPhase = null;
  for (const line of content.split(/\r?\n/)) {
    const phaseMatch = line.match(/^##\s+Phase\s+(\d+).*?—\s*(✅|🚧|⬚)/);
    if (phaseMatch) {
      const phaseId = parseInt(phaseMatch[1]);
      const statusEmoji = phaseMatch[2];
      currentPhase = `Phase ${phaseId}`;
      statuses[currentPhase] = {
        phaseStatus: statusEmoji === '✅' ? 'complete' :
                     statusEmoji === '🚧' ? 'in-progress' : 'planned',
        lessons: {}
      };
      continue;
    }
    if (currentPhase) {
      const lessonMatch = line.match(/^\|\s*\d+\s*\|\s*(.+?)\s*\|\s*(✅|🚧|⬚)\s*\|/);
      if (lessonMatch) {
        const lessonName = lessonMatch[1].trim();
        const statusEmoji = lessonMatch[2];
        const status = statusEmoji === '✅' ? 'complete' :
                       statusEmoji === '🚧' ? 'in-progress' : 'planned';
        statuses[currentPhase].lessons[lessonName] = status;
      }
    }
  }
  return statuses;
}

This function returns an object mapping phase names to their statuses and lesson collections.

Step 2: Merging with README.md Metadata

The parseReadme function (line 338) processes README.md to extract canonical lesson tables containing GitHub URLs. It matches each lesson name to the roadmap entry and enforces consistency. If a lesson has a concrete GitHub link but the roadmap still shows "planned," the script forces the status to complete (lines 70-73):

// site/build.js lines 70-73
if (url && status === 'planned') {
  // Any lesson that has a concrete GitHub link is at least complete.
  status = 'complete';
}

This validation prevents the scenario where a lesson directory exists but the roadmap incorrectly displays it as unstarted.

Step 3: Generating the JSON Output

After assembling the complete phase and lesson objects, the script writes a compact JSON representation to site/data.js (line 209). The writeDataFile function wraps the data in a global variable for browser consumption:

// site/build.js
function writeDataFile(phases) {
  const output = `window.AIFS_DATA = ${JSON.stringify({ phases }, null, 2)};`;
  fs.writeFileSync(path.join(__dirname, 'data.js'), output, 'utf8');
}

The resulting file contains structured arrays with complete metadata including id, name, status, type, lang, and url for every lesson.

Step 4: CI Enforcement and Validation

The site-rebuild workflow runs node site/build.js on every push. If the generated data.js does not reflect the current state of ROADMAP.md (for example, if you manually edited the JSON or forgot to update the markdown glyphs), audit_lessons.py fails the build. This guarantees that the website always mirrors the latest roadmap and eliminates manual copy-and-paste errors.

Why Both Systems Exist

• Human-friendly tracking: Contributors can open ROADMAP.md directly in the GitHub UI to see visual progress indicators without generating the site or running local scripts. The glyphs also feed the roadmap badge displayed on the public site.

• Machine-ready consumption: The React-based front-end requires a deterministic, serializable structure. Auto-generation ensures that URL paths, navigation links, and SEO fields remain synchronized with the actual repository structure. Without this separation, maintaining consistent URLs across 50+ lessons would require manual updates in multiple locations.

Implementation Details

Extracting Status Glyphs

The build script relies on specific regex patterns to identify phases and lessons. Phase headers must match the pattern ## Phase {N} ... — {emoji}, while lesson rows follow the markdown table format | {number} | {title} | {emoji} |. Any deviation from this format causes the parser to skip entries, which is why the CI audit is critical.

Validating Status Consistency

The override logic in parseReadme acts as a safety mechanism. When the script detects a lesson URL in README.md that points to an existing directory, it automatically promotes the status from "planned" to "complete" regardless of the emoji in ROADMAP.md. This ensures the website shows accurate availability even if contributors forget to update the roadmap glyphs immediately after merging a lesson.

Writing the Generated Artifact

The output file site/data.js is a JavaScript module that assigns JSON to window.AIFS_DATA, allowing the static site to load curriculum data without additional fetch requests. This approach minimizes load times for the interactive catalog while maintaining a clear separation between source content and generated assets.

Summary

• ROADMAP.md is the manual source of truth containing visual status glyphs (✅ 🚧 ⬚) for human contributors.
• site/data.js is the auto-generated JSON consumed by the front-end, created by site/build.js.
• The build pipeline parses markdown tables, validates statuses against README.md links, and writes structured data to site/data.js.
• CI enforcement via audit_lessons.py prevents drift between the roadmap and the website.
• Never edit site/data.js manually—always update ROADMAP.md and let the build script regenerate the JSON.

Frequently Asked Questions

What happens if I edit site/data.js directly?

Your changes will be overwritten on the next push. The site-rebuild workflow runs node site/build.js automatically, regenerating the file from ROADMAP.md and README.md. If the audit script detects manual modifications that conflict with the source markdown, the CI job fails and blocks the deployment.

How does the build script handle status mismatches?

The parseReadme function contains validation logic (lines 70-73) that checks if a lesson has a GitHub URL in README.md. If a URL exists but ROADMAP.md shows the lesson as "planned," the script forces the status to "complete." This ensures the website reflects actual repository contents even when the roadmap glyphs lag behind development.

Can I add a new lesson without updating ROADMAP.md?

No. While the lesson content would exist in the repository, the website catalog and navigation structures rely on site/data.js, which is generated exclusively from ROADMAP.md and README.md parsing. Without updating the roadmap table, the build script will not include the lesson in the generated JSON, making it invisible to the site's search and navigation features.

Where is the build automation configured?

The automation lives in two locations: the site/build.js script contains the transformation logic (parseRoadmap, parseReadme, and writeDataFile), while the AGENTS.md file defines the site-rebuild GitHub Actions workflow that triggers the build on every push. The audit_lessons.py script provides the final validation gate to ensure data consistency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}