How site/build.js Automates the AI Curriculum Website for AI Engineering from Scratch

site/build.js automates the AI curriculum website by executing a 14-stage pipeline that parses Markdown curriculum sources, discovers reusable artifacts, synchronizes repository statistics, and generates SEO assets like data.js, sitemap.xml, and llms.txt.

The rohitg00/ai-engineering-from-scratch repository relies on site/build.js as its single source of truth for website generation. This Node.js script eliminates manual updates by transforming raw curriculum Markdown into structured runtime data, ensuring the live site always reflects the current repository state without manual HTML edits.

Curriculum Data Ingestion and Parsing

The automation begins by reading and parsing the core curriculum Markdown files to establish the data foundation.

Parsing ROADMAP.md for Lesson Status

The parseRoadmap() function (lines 31-60) reads the ROADMAP.md file and translates lesson-status emojis into a structured lookup table. It maps ✅ to "complete", 🚧 to "in-progress", and ⬚ to "planned", creating a hierarchical object of {Phase → {lesson → status}} that feeds into the curriculum visualization.

Extracting Phase Structure from README.md

The parseReadme() function (lines 63-131) extracts phase metadata (id, name, description) and lesson tables (name, type, language, link) from README.md. This function merges the status information derived from the roadmap parser, ensuring the frontend receives a unified view of curriculum structure and completion state.

Compiling Glossary Definitions

The parseGlossary() function (lines 71-84) processes glossary/terms.md to generate an array of term objects. Each entry contains the term, what it "says" (definition), and what it "means" (explanation), which powers the site's glossary page search and display features.

Artifact Discovery and Metadata Enrichment

Beyond static Markdown, the script performs deep repository traversal to discover reusable learning materials and enrich lesson data with remote content.

Walking Output Directories for Reusable Artifacts

The discoverArtifacts() function (lines 112-177) recursively walks every phases/*/*/outputs/ directory. It reads front-matter from skill-*.md, prompt-*.md, and agent-*.md files, along with mission.md documentation, assembling a flat list of reusable "artifacts" available for student reference.

Fetching Remote Lesson Summaries

The extractLessonMeta() function (lines 33-68) processes lessons that contain GitHub URLs. For each valid lesson link, it fetches the corresponding docs/en.md file and programmatically extracts a one-line summary and concatenates all H3 headings into a keywords list. This enriches the curriculum data with searchable metadata without requiring manual summary entry.

Static Asset Generation and Serialization

After parsing and enrichment, the script serializes the collected data into JavaScript modules and SEO-critical XML files.

Generating site/data.js

The output generation block (lines 68-78) serializes the complete curriculum object (PHASES), glossary array (GLOSSARY), and artifacts collection (ARTIFACTS) into site/data.js. This single JavaScript module serves as the runtime data layer consumed by the frontend, enabling dynamic rendering of the curriculum catalog without a backend database.

Capturing Build Metadata

The writeBuildMeta() function (lines 100-117) captures the current Git ref (branch or commit SHA) used during the build. This metadata allows the client-side code to fetch the correct version of lesson documentation and display version information to users.

Creating SEO sitemap.xml

The writeSitemap() function (lines 88-107) generates a comprehensive sitemap.xml file covering the home page, catalog, glossary, and every individual lesson URL. This automation ensures search engines discover new lessons immediately upon deployment without manual sitemap updates.

Producing llms.txt for AI Agents

The writeLlms() function (lines 113-139) creates a plain-text, link-rich map of the entire curriculum at site/llms.txt. This file is specifically designed for LLM agents and AI tools to browse the curriculum structure, providing a machine-readable index of all learning resources.

Repository Synchronization and Statistics

The build script maintains consistency between the source repository and the generated website by updating documentation and statistics.

Computing Curriculum Statistics

The statistics computation block (lines 52-66) aggregates totals for phases, lessons, completed lessons, extracted summaries, glossary terms, and discovered artifacts. These metrics drive both the website UI and the llms.txt marketing content.

Syncing README Badges and Traffic Stats

The syncReadme() function (lines 42-80) updates the lesson count badge in README.md and injects a live traffic statistics block from site/stats.json. This ensures the repository documentation displays current metrics automatically.

Updating Marketing Counts in HTML

The syncCounts() function (lines 84-94) searches through static HTML files (index.html, catalog.html, etc.) and replaces hard-coded numerical placeholders with current lesson, phase, and artifact totals. This keeps marketing copy accurate without manual editing.

Executing the Build Pipeline

The entire automation is orchestrated by the build() function (line 602), which executes all stages sequentially and prints progress logs to the console. The script runs automatically on every push via the GitHub Actions workflow defined in curriculum.yml.

Running the Build Locally

Execute the script from the repository root to generate all assets manually:

node site/build.js

The console emits step-by-step progress indicators (e.g., "🔍 Parsing README.md…", "✅ Generated site/data.js").

Consuming Generated Data in the Frontend

The generated site/data.js exports structured curriculum data for import:

import { PHASES, GLOSSARY, ARTIFACTS } from '/site/data.js';

console.log('Available phases:', PHASES);
console.log('Glossary terms:', GLOSSARY);
console.log('Reusable artifacts:', ARTIFACTS);

Inspecting Generated SEO Files

Verify sitemap generation and preview the LLM-friendly curriculum map:

# Preview the first 15 lines of the LLM curriculum map

cat site/llms.txt | head -n 15

# List all URLs in the sitemap

cat site/sitemap.xml | grep '<loc>'

Summary

• site/build.js serves as the single automation source for the AI Engineering from Scratch curriculum website, located in the rohitg00/ai-engineering-from-scratch repository.
• The script parses README.md, ROADMAP.md, and glossary/terms.md to construct a unified curriculum data model.
• It discovers reusable artifacts by walking phases/*/*/outputs/ directories and reading front-matter from skill and prompt files.
• Generated outputs include site/data.js (frontend data), sitemap.xml (SEO), and llms.txt (AI agent consumption).
• Repository synchronization functions keep README.md badges and static HTML marketing counts in perfect sync with actual repository statistics.
• The build runs automatically via GitHub Actions or manually via node site/build.js.

Frequently Asked Questions

What is the primary purpose of site/build.js?

site/build.js is the build automation script that transforms static Markdown curriculum files into the dynamic data layer and SEO assets required by the AI Engineering from Scratch website. It eliminates manual website updates by parsing repository content and generating runtime files like data.js, sitemap.xml, and llms.txt on every deployment.

How does site/build.js handle curriculum updates?

The script automatically reflects curriculum changes by re-parsing README.md and ROADMAP.md during each build execution. When lessons are added, removed, or status emojis are updated in the roadmap, the parseReadme() and parseRoadmap() functions capture these changes and regenerate the data structures without requiring manual code changes.

What files does site/build.js generate?

The automation generates four primary file types: site/data.js containing the structured curriculum data (PHASES, GLOSSARY, ARTIFACTS), site/sitemap.xml for search engine optimization, site/llms.txt as a plain-text curriculum map for AI agents, and updated inline statistics within README.md and static HTML files.

Can I run site/build.js outside of GitHub Actions?

Yes, site/build.js is a standard Node.js script that executes locally. Run node site/build.js from the repository root to generate all assets manually, which is useful for local development, debugging SEO files, or previewing curriculum changes before pushing to production.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}