How site/build.js Processes README and ROADMAP Files into data.js

The site/build.js script parses curriculum markdown from README.md and ROADMAP.md, extracts structured metadata about phases and lessons, and compiles a static JavaScript module at site/data.js that powers the AI Engineering from Scratch website frontend.

The ai-engineering-from-scratch repository maintains its curriculum documentation in plain markdown files while serving a dynamic web interface. The site/build.js file functions as a deterministic compiler that bridges these two layers, transforming human-readable documentation into machine-consumable data structures without manual intervention.

Path Setup and Input Resolution

The script begins by declaring absolute paths to the source files it will consume. These constants anchor the parser to the repository root and ensure consistent file resolution across environments.

const REPO_ROOT = path.resolve(__dirname, '..');
const README_PATH      = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH     = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH    = path.join(REPO_ROOT, 'glossary', 'terms.md');
const OUTPUT_PATH      = path.join(__dirname, 'data.js');

Lines 15‑20 of site/build.js define these paths, establishing the input corpus that feeds the build pipeline. The script treats README.md as the canonical curriculum structure, ROADMAP.md as the status authority, and glossary/terms.md as a supplemental knowledge base.

Parsing ROADMAP.md for Lesson Statuses

The parseRoadmap function (lines 101‑133) implements a line‑by‑line state machine that converts emoji‑based progress indicators into normalized strings. It recognizes three completion states: complete (✅), in‑progress (🚧), and planned (⬚).

function parseRoadmap(content) {
  const statuses = {};
  let currentPhase = null;
  
  for (const line of content.split(/\r?\n/)) {
    // Phase header: "## Phase 0: ... — ✅"

    const phaseMatch = line.match(/^##\s+Phase\s+(\d+).*?—\s*(✅|🚧|⬚)/);
    
    // Lesson rows: "| 01 | Dev Environment | ✅ |"
    const lessonMatch = line.match(/^\|\s*\d+\s*\|\s*(.+?)\s*\|\s*(✅|🚧|⬚)\s*\|/);
  }
  return statuses;
}

This function produces a nested map structure where each phase contains its aggregated status and individual lesson states. The regex patterns specifically target the markdown table syntax used in ROADMAP.md to extract ordinal numbers and completion emojis.

Processing README.md for Curriculum Structure

The parseReadme function (lines 138‑210) performs the heavy lifting of curriculum extraction. It scans README.md for phase headers using multiple pattern matchers to accommodate both legacy and modern markdown formatting, including badge‑based syntaxes and collapsible details blocks.

Once the parser identifies a phase boundary, it enters a table‑scanning mode to extract lesson metadata. The function splits pipe‑delimited table rows to capture four critical fields: lesson name, type, programming language, and optional URL.

if (inLessonTable && currentPhase && line.startsWith('|')) {
  const cols = line.split('|').map(c => c.trim()).filter(c => c.length > 0);
  if (cols.length >= 4) {
    const lessonEntry = {
      name: lessonName.trim(),
      status,
      type: isCapstoneTable ? 'Capstone' : type.trim(),
      lang: lang.trim() || '—',
      ...(url && { url })
    };
    currentPhase.lessons.push(lessonEntry);
  }
}

The parser performs a fuzzy match against the roadmap data to resolve lesson statuses, automatically promoting any lesson with a valid hyperlink to complete status regardless of the roadmap emoji.

Aggregating Auxiliary Data Sources

Beyond the core curriculum files, build.js incorporates two additional data layers. The glossary is loaded as raw JSON (approximately line 1200) and injected unchanged into the final payload. Meanwhile, parseLearningPaths (lines 308‑447) traverses the learning-paths/ directory, reading thematic JSON overlays that augment the base phase structure with alternative navigation routes and validated prerequisite graphs.

Compiling and Writing the Output Module

With all data structures populated, the script assembles a JavaScript module template using template literals with pretty‑printed JSON serialization. This approach maintains human readability in the generated output while ensuring valid JavaScript syntax.

const output = `// Auto‑generated by build.js — do not edit manually.
const ROADMAP_PREREQS = ${JSON.stringify(roadmapPrereqs, null, 2)};
const PHASES         = ${JSON.stringify(phases, null, 2)};
const LEARNING_PATHS = ${JSON.stringify(learningPaths, null, 2)};
const GLOSSARY       = ${JSON.stringify(glossaryTerms, null, 2)};
`;

Lines 2081‑2095 construct this payload, and lines 2097‑2099 persist it to site/data.js using fs.writeFileSync. The resulting file becomes the single source of truth for the frontend, exported as named constants that the client bundle imports to render phase tables, lesson cards, and navigation elements.

Generating SEO and Discovery Artifacts

Following the primary data compilation, the script executes auxiliary generators that produce lesson-seo.json, certification-seo.json, figure-manifest.js, and sitemap.xml. These functions—such as writeSeoArtifacts (lines 1181‑1199) and writeFigureManifest (lines 628‑635)—ensure that curriculum changes automatically propagate to search engine metadata and static HTML discovery pages.

Summary

• site/build.js serves as a markdown‑to‑JavaScript compiler that eliminates manual data entry between documentation and website.
• The parseRoadmap function normalizes emoji status indicators (✅ 🚧 ⬚) into structured strings while scanning ROADMAP.md tables.
• parseReadme extracts phase hierarchies and lesson metadata from markdown tables in README.md, supporting multiple header formats and linked lesson references.
• The script aggregates glossary terms and learning path overlays to produce a comprehensive curriculum dataset.
• Output occurs at lines 2097‑2099, writing a pretty‑printed JavaScript module to site/data.js that the frontend imports directly.

Frequently Asked Questions

What input files does site/build.js require to generate data.js?

The script requires three primary inputs located in the repository root: README.md for curriculum structure, ROADMAP.md for completion statuses, and glossary/terms.md for terminology definitions. It also scans the learning-paths/ directory for optional overlay JSON files that provide alternative navigation routes through the curriculum.

How does the build script determine if a lesson is complete?

The parseRoadmap function uses regex to capture emoji characters (✅, 🚧, or ⬚) from table rows in ROADMAP.md, mapping them to complete, in-progress, or planned respectively. Additionally, the parseReadme function automatically assigns complete status to any lesson that includes a valid markdown hyperlink, assuming that linked content represents finished material.

Can I run site/build.js without building the entire website?

Yes. Executing node site/build.js from the repository root runs the compiler independently of the website bundler. The script outputs site/data.js along with auxiliary SEO and manifest files, making it suitable for CI/CD pipelines that need to validate curriculum data or regenerate static assets without triggering a full frontend build.

What other artifacts does build.js produce besides data.js?

In addition to the primary curriculum module, the script generates lesson-seo.json and certification-seo.json for search engine metadata, figure-manifest.js for mapping figure providers, and sitemap.xml for crawler discovery. It also updates static HTML fragments in catalog.html and certifications.html to reflect the latest lesson availability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}