How site/build.js Generates data.js from README.md and ROADMAP.md

TLDR: The site/build.js script in the rohitg00/ai-engineering-from-scratch repository parses markdown tables from README.md and ROADMAP.md to extract phase and lesson structures, enriches them with per-lesson metadata and glossary terms, and writes a static JavaScript module to site/data.js that exports PHASES, GLOSSARY, and ARTIFACTS arrays for the curriculum site.

The static site powering the AI Engineering From Scratch curriculum doesn't rely on a traditional CMS. Instead, site/build.js generates data.js from README.md and ROADMAP.md to create a single source of truth where documentation changes automatically synchronize with the website's navigation and progress tracking.

Loading the Source Documentation

The build process starts by resolving paths relative to the repository root. In site/build.js, the script defines constants pointing to the primary curriculum files (lines 23‑26):

const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

These paths enable the script to read the human-readable curriculum definition alongside the glossary terms located in glossary/terms.md.

Parsing ROADMAP.md for Lesson Status

The parseRoadmap() function (lines 30‑61) serves as the single source of truth for lesson completion status. It scans ROADMAP.md line-by-line to extract:

• Phase headings marked with status emojis: ✅ (complete), 🚧 (in‑progress), or ⬚ (planned)
• Lesson rows from markdown tables formatted as │ 01 │ Dev Environment │ ✅ │

The function returns a nested object mapping each phase to its status and a dictionary of lesson names to their respective statuses (complete, in‑progress, or planned).

Extracting Curriculum Structure from README.md

While ROADMAP.md tracks status, parseReadme() (lines 63‑131) extracts the hierarchical curriculum structure from README.md. This function identifies phase headers—supporting both the legacy "### Phase 0:" format and newer badge-style headers—and initializes currentPhase objects containing id, name, and status pulled from the roadmap data.

When the parser encounters lesson tables (detecting rows starting with │ # │ Lesson), it extracts:

• Lesson name and optional URL from the lesson column
• Lesson type (Build or Learn) by stripping badge images
• Programming languages by converting emoji flags to plain text using the internal EMOJI_LANG mapping

Each lesson object receives its status by matching against the roadmap data, with one exception: lessons containing a URL are automatically forced to complete status. The completed lesson objects are pushed into currentPhase.lessons arrays, ultimately returning the full phases array.

Enriching Lessons with Metadata

After establishing the curriculum tree, the build loop (lines 39‑49) walks every lesson that has a defined URL and calls extractLessonMeta() (lines 33‑68). This function reads each lesson's docs/en.md file to extract:

• A summary from the first blockquote (one-liner description)
• Keywords concatenated from all ### headings within the lesson document

These fields populate the summary and keywords properties on each lesson object, enabling search and preview functionality on the frontend.

Aggregating Glossary and Reusable Artifacts

Beyond the core curriculum, the script aggregates supplementary content through two dedicated functions:

parseGlossary() (lines 71‑103) scans glossary/terms.md for ### headings and extracts "What people say / What it actually means" pairs, returning an array of term objects that become the GLOSSARY export.

discoverArtifacts() (lines 40‑98) walks the phases/ directory to discover reusable outputs. It identifies files in outputs/ subdirectories matching patterns skill-*.md, prompt-*.md, or agent-*.md, parses their frontmatter, and includes any mission.md files found in lesson directories. These are compiled into the ARTIFACTS export for quick reference.

Writing the data.js Module

With phases, glossaryTerms, and artifacts collected, the script constructs the final output string. It writes to site/data.js (lines 69‑78) a JavaScript module prefixed with an auto-generation warning comment:

// Auto-generated by build.js — do not edit manually.
const PHASES = [...];
const GLOSSARY = [...];
const ARTIFACTS = [...];

This file serves as the static data layer for the curriculum website, consumed directly by the frontend to render phase navigation, lesson lists, and search indexes.

Auxiliary Build Tasks

The build() function (lines 19‑86) orchestrates additional synchronization tasks executed at build time:

• syncReadme(): Updates completion badge counts in README.md
• syncCounts(): Synchronizes marketing numbers across documentation
• writeSitemap(): Generates XML sitemaps for SEO
• LLMs.txt generation: Creates an AI-friendly digest of the curriculum
• Build metadata: Stores the current Git ref in build-meta.js

The entire pipeline executes when the script runs via build(); on line 601, typically triggered by GitHub Actions on every commit.

Running the Build Locally

Execute the generator from the repository root to regenerate site/data.js:

node site/build.js

The script prints progress logs and overwrites site/data.js with the latest curriculum state derived from the current markdown files.

Consuming the Generated Data

Import the generated constants in frontend components or Node.js scripts:

import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// Filter for completed Python lessons
const pythonLessons = PHASES.flatMap(p => p.lessons)
  .filter(l => l.status === 'complete' && l.lang.includes('Python'));

Summary

• site/build.js transforms markdown documentation into structured JavaScript by parsing README.md for curriculum hierarchy and ROADMAP.md for completion status
• The parseRoadmap() and parseReadme() functions extract phase structures, lesson metadata, and language mappings from markdown tables
• extractLessonMeta() enriches lessons with summaries and keywords by reading individual lesson docs/en.md files
• The script aggregates glossary terms from glossary/terms.md and reusable artifacts from phases/ subdirectories via parseGlossary() and discoverArtifacts()
• Output is written to site/data.js as static exports (PHASES, GLOSSARY, ARTIFACTS) consumed by the curriculum website

Frequently Asked Questions

How does the build script determine a lesson's completion status?

The script checks ROADMAP.md for emoji indicators (✅ for complete, 🚧 for in-progress, ⬚ for planned) matched to lesson names. Additionally, any lesson with a defined URL in README.md is automatically forced to complete status regardless of the roadmap indicator.

Can I modify the generated data.js file manually?

No. The file header explicitly states // Auto-generated by build.js — do not edit manually. Any manual changes will be overwritten the next time node site/build.js executes, which typically occurs automatically via GitHub Actions on every repository push.

What markdown structure does the parser expect for lesson tables?

The script expects standard GitHub-flavored markdown tables with columns for lesson number, name, type (Build/Learn badges), and languages (emoji flags). The parser specifically looks for rows starting with │ (pipe) characters and extracts cells based on the │ # │ Lesson header pattern found in README.md.

Where does the glossary content originate?

Glossary entries are sourced from glossary/terms.md, where the parseGlossary() function extracts ### headings as terms and parses "What people say / What it actually means" lines as definitions. This content is exported as the GLOSSARY constant in site/data.js.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}