How site/build.js Generates data.js from README.md and ROADMAP.md in ai-engineering-from-scratch

The site/build.js script reads README.md and ROADMAP.md, parses their markdown tables and status indicators, enriches the data with lesson metadata and glossary terms, and exports a JavaScript module to site/data.js that powers the curriculum website.

The ai-engineering-from-scratch repository uses a custom build pipeline to transform static markdown documentation into a structured JavaScript data module. Understanding how site/build.js generates data.js from README.md and ROADMAP.md reveals how the curriculum site maintains synchronization with the source documentation without manual data entry. This automated process ensures that lesson statuses, phase structures, and glossary definitions always reflect the current state of the repository.

Build Pipeline Overview

The build system orchestrates multiple parsing stages to convert human-readable markdown into machine-readable JavaScript constants. Located at site/build.js, the script executes an eight-step pipeline that processes curriculum files, extracts metadata, and writes the final output to site/data.js. Each function handles a specific data transformation, from parsing roadmap status indicators to discovering reusable artifacts across lesson directories.

Step 1: Loading Source Files

The script begins by resolving absolute paths to the repository root and target markdown files. According to lines 23–26 of site/build.js, it defines constants for the three primary inputs:

const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

These paths enable the build system to read README.md for curriculum structure, ROADMAP.md for completion status, and glossary/terms.md for domain definitions.

Step 2: Parsing ROADMAP.md for Lesson Status

The parseRoadmap() function (lines 30–61) processes the roadmap file to extract phase headings and lesson completion states. It scans each line to identify:

• Phase headings marked with status emojis (✅ for complete, 🚧 for in-progress, ⬚ for planned)
• Lesson rows formatted as markdown tables (| 01 | Dev Environment | ✅ |)

The function returns a nested object mapping phases to their statuses and lessons, creating a single source of truth for curriculum completion states.

Step 3: Parsing README.md for Curriculum Structure

The parseReadme() function (lines 63–131) transforms the README's markdown tables into structured phase objects. It detects phase headers (both legacy "### Phase 0:" formats and newer badge styles) and initializes currentPhase objects with id, name, and status properties inherited from the roadmap data.

When encountering lesson tables (starting with | # | Lesson …), the parser extracts:

• Lesson names and optional URLs from the lessonCol field
• Type classifications (Build or Learn) by stripping badge images
• Programming languages through emoji-to-text conversion using the EMOJI_LANG mapping

Each lesson is matched against the roadmap data to assign complete, in-progress, or planned status. If a lesson includes a URL, the status is automatically set to complete.

Step 4: Extracting Per-Lesson Metadata

For lessons containing URLs, the build script calls extractLessonMeta() (lines 33–68) to augment curriculum data with content summaries. This function reads each lesson's docs/en.md file and extracts:

• Summary text from the first blockquote (used as a one-line description)
• Keywords concatenated from all ### headings within the lesson

The build loop (lines 39–49) integrates this metadata into the lesson objects, producing enriched entries containing summary and keywords fields alongside the core curriculum data.

Step 5: Collecting Glossary Terms and Reusable Artifacts

The pipeline extends beyond curriculum content to include supplementary resources. The parseGlossary() function (lines 71–103) scans glossary/terms.md for ### headings and "What people say / What it actually means" lines to build the GLOSSARY array.

Simultaneously, discoverArtifacts() (lines 40–98) walks the phases/ directory to locate reusable outputs:

• Skill files matching outputs/skill-*.md
• Prompt files matching outputs/prompt-*.md
• Agent files matching outputs/agent-*.md
• Mission files named mission.md within lesson directories

Each artifact's frontmatter is parsed to populate the ARTIFACTS export.

Step 6: Generating the Final data.js File

After aggregating phases, glossaryTerms, and artifacts, the script constructs the final output string. Lines 69–78 write a JavaScript module containing three exported constants:

const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

This content is written to site/data.js with an auto-generated comment header warning against manual edits. The resulting file serves as the data layer for the curriculum website, consumed by downstream components to render the dynamic interface.

Running the Build Locally

Execute the build script from the repository root to regenerate the data module:

node site/build.js

The script prints progress logs and recreates site/data.js with the latest curriculum state. Import the generated module in your application to access the structured data:

import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// List all completed lessons
const completed = PHASES.flatMap(p => p.lessons)
                       .filter(l => l.status === 'complete')
                       .map(l => l.name);
console.log('Completed lessons:', completed);

Inspect the generated structure to verify the output:

head -n 20 site/data.js

The output begins with an auto-generated warning followed by the PHASES array containing the complete curriculum hierarchy.

Summary

• site/build.js serves as the central build pipeline that transforms markdown documentation into a JavaScript data module.
• The script parses ROADMAP.md (lines 30–61) to extract phase and lesson completion statuses using emoji indicators.
• README.md parsing (lines 63–131) extracts curriculum structure, lesson types, and programming languages from markdown tables.
• Per-lesson metadata is enriched by reading docs/en.md files via extractLessonMeta() to capture summaries and keywords.
• The final site/data.js exports three constants—PHASES, GLOSSARY, and ARTIFACTS—that power the curriculum website.
• Running node site/build.js locally regenerates the data file, ensuring the site always reflects the current repository state.

Frequently Asked Questions

Where does site/build.js look for source files?

The script resolves paths relative to its own directory, looking for README.md, ROADMAP.md, and glossary/terms.md in the repository root. It defines these paths using path.resolve(__dirname, '..') to ensure cross-platform compatibility.

How does the build script determine if a lesson is complete?

The script checks ROADMAP.md for status emojis (✅, 🚧, or ⬚) in the lesson table rows. Additionally, if a lesson entry in README.md contains a URL field, the parseReadme() function automatically assigns complete status regardless of the roadmap indicator.

Can I modify data.js manually?

No. The file header explicitly states that site/data.js is auto-generated by build.js and should not be edited manually. Any changes would be overwritten the next time the build script runs, either locally or via the GitHub Actions workflow.

What data structures are exported from data.js?

The generated module exports three constants: PHASES (an array of curriculum phases containing nested lesson objects), GLOSSARY (an array of term definitions), and ARTIFACTS (an array of reusable skills, prompts, and agents with their frontmatter metadata).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}