Site Build Process for Generating data.js from README and ROADMAP Files

The site build process uses site/build.js to parse README.md and ROADMAP.md, extracting curriculum structure and lesson statuses to generate site/data.js, a JavaScript module that powers the static curriculum site.

The rohitg00/ai-engineering-from-scratch repository automates its curriculum website through a Node.js build pipeline. This system transforms human-readable markdown documentation into structured JavaScript data, eliminating manual synchronization between repository documentation and the deployed site.

How the Build Script Reads Source Files

The build process begins in site/build.js by resolving paths to the repository's core documentation files. At lines 23-26, the script defines constants pointing to README.md, ROADMAP.md, and glossary/terms.md in the repository root:

const REPO_ROOT   = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');

These paths serve as the single source of truth for curriculum content, status tracking, and terminology definitions.

Parsing ROADMAP.md for Lesson Status

The parseRoadmap() function (lines 30-61) scans ROADMAP.md to extract the completion status of every lesson and phase. It processes the file line-by-line to identify:

• Phase headings marked with status emojis (✅ for complete, 🚧 for in-progress, ⬚ for planned)
• Lesson rows formatted as tables (| 01 | Dev Environment | ✅ |)

The function returns a nested object mapping phase names to their status and lesson completion states, enabling the build to determine which lessons are complete, in-progress, or planned based on the emoji indicators present in the roadmap.

Extracting Curriculum Structure from README.md

After establishing lesson statuses, the parseReadme() function (lines 63-131) processes README.md to build the curriculum hierarchy. This function:

1. Detects phase headers using both legacy "### Phase 0:" formats and newer badge-style headers

2. Identifies lesson tables by scanning for | # | Lesson … patterns

3. Extracts lesson metadata including:

   • Lesson name and optional URL from the lesson column
   • Type (Build or Learn) by stripping badge images
   • Languages by converting emoji flags to plain text using the EMOJI_LANG mapping

Each lesson object is matched against the roadmap data to assign its status. If a lesson includes a URL in README.md, the build forces its status to complete regardless of the roadmap indicator. The function returns a phases array containing phase objects with id, name, status, and a lessons array.

Enriching Data with Per-Lesson Metadata

Once the basic structure is established, the build augments each lesson with detailed metadata using extractLessonMeta() (lines 33-68). For every lesson that has a URL, the function:

• Reads the corresponding docs/en.md file from the lesson directory
• Extracts the first blockquote as a one-line summary
• Concatenates all ### headings to form a keywords string

The build loop (lines 39-49) iterates through all lessons, calling this extraction function and attaching the resulting summary and keywords properties to the lesson objects.

Collecting Glossary Terms and Reusable Artifacts

The build aggregates supplementary content from two additional sources:

Glossary Terms: The parseGlossary() function (lines 71-103) scans glossary/terms.md for ### headings and "What people say / What it actually means" lines, returning an array of term objects for the GLOSSARY export.

Reusable Artifacts: The discoverArtifacts() function (lines 40-98) walks the phases/ directory, searching for outputs/*.md files prefixed with skill-, prompt-, or agent-. It parses front-matter from these files and also discovers mission.md files in lesson directories, compiling them into the ARTIFACTS array.

Generating the Final data.js File

With phases, glossaryTerms, and artifacts collected, the script constructs the final output at lines 69-78. It writes a JavaScript module to site/data.js containing three exported constants:

// Auto-generated by build.js — do not edit manually.
const PHASES   = [...];   // full curriculum tree
const GLOSSARY = [...];   // glossary entries
const ARTIFACTS = [...];  // reusable outputs

The file is prefixed with a comment indicating it is auto-generated, preventing manual edits that would be overwritten by subsequent builds.

Auxiliary Build Tasks

Beyond generating data.js, the build() function (lines 19-86) orchestrates several maintenance tasks:

• Synchronizing README badges via syncReadme() to update completion counts
• Updating marketing numbers via syncCounts()
• Generating SEO files including sitemap.xml via writeSitemap() and llms.txt for AI consumption
• Recording build metadata by storing the Git ref in build-meta.js

These tasks ensure the repository documentation and auxiliary files remain consistent with the actual lesson content.

Running the Build Locally

To execute the site build process manually or test changes before committing:

# From the repository root

node site/build.js

This command prints progress logs and regenerates site/data.js. You can verify the output by inspecting the generated file:

# Show the first 20 lines of the generated module

head -n 20 site/data.js

To consume the generated data in a site page:

import { PHASES, GLOSSARY, ARTIFACTS } from './site/data.js';

// List all completed lessons
const completed = PHASES.flatMap(p => p.lessons)
                       .filter(l => l.status === 'complete')
                       .map(l => l.name);
console.log('Completed lessons:', completed);

Summary

• The build script site/build.js serves as the central pipeline that transforms markdown documentation into structured JavaScript data.
• Source files README.md, ROADMAP.md, and glossary/terms.md provide the raw content, while the phases/ directory contains reusable artifacts.
• Parsing functions parseRoadmap(), parseReadme(), and extractLessonMeta() extract status, structure, and metadata respectively.
• Output generation creates site/data.js exporting PHASES, GLOSSARY, and ARTIFACTS constants for the website to consume.
• Automation via GitHub Actions ensures the site stays synchronized with repository changes without manual data entry.

Frequently Asked Questions

What triggers the site build process for generating data.js?

The build executes automatically via GitHub Actions workflows when changes are pushed to the repository, or manually by running node site/build.js from the repository root. This ensures site/data.js always reflects the current state of README.md and ROADMAP.md.

How does the build determine a lesson's completion status?

The parseRoadmap() function checks for emoji indicators in ROADMAP.md (✅ for complete, 🚧 for in-progress, ⬚ for planned). Additionally, if a lesson row in README.md contains a URL, the build forces the status to complete regardless of the roadmap indicator.

Why is data.js auto-generated instead of maintained manually?

Auto-generation ensures a single source of truth where README.md and ROADMAP.md remain the authoritative curriculum documents. Manual maintenance would risk divergence between the human-readable documentation and the website data, whereas the build script guarantees consistency.

What data structures does the generated data.js export?

The file exports three constants: PHASES (the curriculum tree with lesson metadata), GLOSSARY (term definitions from glossary/terms.md), and ARTIFACTS (reusable skills, prompts, and agents from the phases/ directory).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}