How the Site Build System Generates data.js from README and ROADMAP in AI Engineering From Scratch

The site build system in ai-engineering-from-scratch uses a Node.js script at site/build.js to parse structured markdown tables from README.md and ROADMAP.md, extracting lesson metadata, phase statuses, and glossary terms to auto-generate the static site/data.js module that powers the curriculum website.

The ai-engineering-from-scratch repository maintains its curriculum as human-readable markdown documentation. Instead of manually maintaining a separate data file, the project uses an automated pipeline that transforms these markdown sources into a machine-readable JavaScript module. This approach ensures the website curriculum stays perfectly synchronized with the repository's learning roadmap.

Overview of the Build Pipeline

The core build logic lives in [site/build.js](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/build.js) and executes automatically via GitHub Actions or manually via Node.js. The script begins by resolving paths to the repository root and loading three primary source files:

const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

When invoked, the build() function (lines 19‑86) orchestrates a multi-stage pipeline that reads these files, parses their structured content, enriches the data with additional metadata, and writes the final output to site/data.js.

Parsing ROADMAP.md for Lesson Status

The parseRoadmap() function (lines 30‑61) serves as the single source of truth for lesson completion status. It processes ROADMAP.md line-by-line to extract phase headings and lesson rows.

The function identifies:

• Phase status via emoji indicators: ✅ = complete, 🚧 = in-progress, ⬚ = planned
• Lesson entries from table rows matching the pattern | 01 | Dev Environment | ✅ |

It returns a nested object structure mapping each phase to its status and lessons, enabling downstream functions to assign accurate status values to curriculum items.

Extracting Curriculum Structure from README.md

The parseReadme() function (lines 63‑131) handles the heavy lifting of curriculum extraction from README.md. This function supports both legacy "Phase N:" headers and newer badge-style formatting.

When the parser encounters a phase header, it initializes a currentPhase object with an id, name, and status (pulled from the roadmap data). It then detects the start of lesson tables using the pattern | # | Lesson … and extracts for each row:

• Lesson name and URL from the lessonCol column
• Type classification (Build vs. Learn) by stripping badge images
• Programming languages via emoji-to-text conversion using the EMOJI_LANG mapping

If a lesson includes a URL, the parser forces its status to complete. Otherwise, it matches the lesson name against the roadmap data to determine whether it is complete, in-progress, or planned. Each processed lesson becomes an entry in the currentPhase.lessons array.

Enriching Data with Lesson Metadata

After establishing the curriculum structure, the build system augments each lesson with additional metadata. The extractLessonMeta() function (lines 33‑68) reads individual lesson files from their docs/en.md paths to extract:

• Summary: The first blockquote found in the markdown file
• Keywords: Concatenated ### headings from the lesson content

The main build loop (lines 39‑49) calls this function for every lesson that has a URL, adding summary and keywords properties to the lesson objects before they are added to the final phases array.

Collecting Glossary and Artifacts

The build system consolidates auxiliary content through two additional parsers:

parseGlossary() (lines 71‑103) scans glossary/terms.md for ### headings and "What people say / What it actually means" lines, returning an array of term objects for the GLOSSARY export.

discoverArtifacts() (lines 40‑98) traverses the phases/ directory searching for outputs/*.md files with specific prefixes (skill-, prompt-, agent-). It parses their frontmatter and also identifies mission.md files within lesson directories, building a flat ARTIFACTS list of reusable outputs.

Generating the Final data.js File

With all data collected, the script constructs the final JavaScript module. It builds a string containing three exported constants:

const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

This string is written to [site/data.js](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) (lines 69‑78) with an auto-generated header comment warning against manual edits. The file also triggers auxiliary tasks including syncReadme (badge count updates), syncCounts (marketing numbers), writeSitemap, and llms.txt generation.

Running the Build Locally

You can execute the build process manually to regenerate site/data.js or inspect the output:

# From the repository root

node site/build.js

# → prints progress logs and (re)creates site/data.js

To inspect the generated structure:

# Show the first 20 lines of the generated file

head -n 20 site/data.js

The output begins with an auto-generated marker:

// Auto-generated by build.js — do not edit manually.
const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    // ...
  }
]

Summary

• The site/build.js script is the central pipeline that transforms markdown documentation into JavaScript data structures.
• parseRoadmap() extracts lesson completion statuses (✅/🚧/⬚) from ROADMAP.md tables.
• parseReadme() parses README.md for phase hierarchies, lesson URLs, types, and languages, then maps them to roadmap statuses.
• extractLessonMeta() enriches lessons with summaries and keywords from individual docs/en.md files.
• The final output at site/data.js exports three constants—PHASES, GLOSSARY, and ARTIFACTS—that power the entire curriculum website without manual data entry.

Frequently Asked Questions

What triggers the site build process?

The build executes automatically when GitHub Actions detects changes to the repository, or manually by running node site/build.js from the repository root. The workflow ensures site/data.js remains synchronized with any edits to README.md or ROADMAP.md.

How does the system determine if a lesson is complete?

The build system checks ROADMAP.md for emoji indicators (✅ for complete, 🚧 for in-progress, ⬚ for planned). Additionally, if parseReadme() detects a URL in the lesson table row, it automatically assigns complete status regardless of the roadmap marker.

Can I manually edit the generated site/data.js file?

No. The file header explicitly states it is auto-generated by build.js. Manual edits would be overwritten during the next build cycle. Instead, modify the source files (README.md, ROADMAP.md, or lesson docs/en.md) and rerun the build script.

What other files does the build system generate?

Beyond site/data.js, the build() function creates site/build-meta.js (containing Git ref information), updates README badge counts via syncReadme, maintains marketing statistics through syncCounts, generates a sitemap via writeSitemap, and produces an llms.txt file for AI consumption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}