How site/build.js Generates site/data.js from the AI Curriculum in ai-engineering-from-scratch

The site/build.js script in rohitg00/ai-engineering-from-scratch parses markdown curriculum files (README.md, ROADMAP.md, and lesson documentation) to generate a static site/data.js bundle containing three serialized JavaScript constants—PHASES, GLOSSARY, and ARTIFACTS—that power the documentation website.

The rohitg00/ai-engineering-from-scratch repository uses a custom Node.js build pipeline to transform its textual curriculum into a machine-readable JavaScript data file. This automated process ensures the static website always reflects the latest lesson statuses, glossary definitions, and reusable code artefacts without requiring manual JSON updates.

Build Pipeline Architecture

The generation process follows a deterministic seven-step pipeline defined in site/build.js. The script establishes absolute paths to key curriculum files, then orchestrates a series of parsing functions that extract structured data from markdown sources.

const REPO_ROOT = path.resolve(__dirname, '..');
const README_PATH = path.join(REPO_ROOT, 'README.md');
const ROADMAP_PATH = path.join(REPO_ROOT, 'ROADMAP.md');
const GLOSSARY_PATH = path.join(REPO_ROOT, 'glossary', 'terms.md');
const OUTPUT_PATH = path.join(__dirname, 'data.js');

Lines 14-19 in site/build.js define the canonical file paths that the build script consumes.

Parsing ROADMAP.md for Status Tracking

The parseRoadmap() function scans ROADMAP.md to extract per-phase and per-lesson completion statuses. It detects phase headers using patterns like ## Phase X … — ✅|🚧|⬚ and parses lesson table rows to map status emojis to structured states.

function parseRoadmap(content) {
  const statuses = {};
  // …iterates line‑by‑line, detecting “## Phase X … — ✅|🚧|⬚” headers

  // and “| 01 | Dev Environment | ✅ |” lesson rows
}

This function returns an object keyed by "Phase <id>" containing the phase status and a nested map of lesson names to their respective states (complete, in-progress, or planned). The implementation occupies lines 31-61 in site/build.js.

Extracting Phase and Lesson Metadata from README.md

The parseReadme() function (lines 63-130) processes the master curriculum list to identify phases, lessons, types, and GitHub URLs. It detects phase headers using both legacy and new badge formats, then extracts lesson tables to capture:

• Lesson name and type (e.g., "Build", "Theory")
• Programming languages (Python, TypeScript, Rust)
• GitHub repository URLs
• Initial status derived from the roadmap data

The function constructs an array of phase objects, each containing an ordered list of lesson objects that serve as the foundation for the PHASES constant.

Enriching Lessons with Summaries and Keywords

After establishing the lesson hierarchy, the script calls extractLessonMeta() for each lesson that contains a GitHub URL. This function reads the lesson's docs/en.md file to extract human-readable metadata.

function extractLessonMeta(relPath) {
  const docPath = path.join(REPO_ROOT, relPath, 'docs', 'en.md');
  // Reads the file, grabs the first “> …” blockquote as `summary`
  // Collects all H3 headings as `keywords`
}

The implementation (lines 33-68) captures the first blockquote as the lesson summary and aggregates all H3 headings as searchable keywords. The main loop (lines 41-49) attaches these summary and keywords properties to each lesson object, enriching the curriculum data with descriptive content for the website's search functionality.

Building the Glossary Index

The parseGlossary() function (lines 71-100) processes glossary/terms.md to create a searchable dictionary of AI terminology. It identifies terms by detecting ### Term headings, then extracts definitions from specific markdown patterns:

• **What people say:** captures common usage
• **What it actually means:** captures technical definitions

The function produces an array of objects with the structure { term, says, means }, which becomes the GLOSSARY constant in the output file.

Discovering Reusable Artefacts

The discoverArtifacts() function (lines 40-98) traverses the phases/*/*/outputs/ directory tree to identify reusable curriculum components. It extracts front-matter from markdown files and registers missions, skills, prompts, and agents as artefacts.

Each artefact record includes:

• kind: The artefact type (skill, prompt, agent, mission)
• name and description: Human-readable identifiers
• tags: Categorical metadata
• phase and lesson: Numeric indices for navigation
• lessonPath and file: Absolute and relative file paths

This discovery process ensures that practical code examples and templates are indexed separately from theoretical content.

Serializing Output to data.js

The build() function (lines 68-78) orchestrates the final output generation. It aggregates the parsed data into three JavaScript constants and writes them to site/data.js as a self-contained module:

function build() {
  // …reads README, ROADMAP, glossary, discovers artefacts…
  const output = `// Auto‑generated by build.js — do not edit manually.
  // Last built: ${new Date().toISOString()}

  const PHASES = ${JSON.stringify(phases, null, 2)};

  const GLOSSARY = ${JSON.stringify(glossaryTerms, null, 2)};

  const ARTIFACTS = ${JSON.stringify(artifacts, null, 2)};
  `;
  fs.writeFileSync(OUTPUT_PATH, output, 'utf8');
}

The generated file uses literal JavaScript syntax rather than JSON, allowing the static site to load the data instantly via a <script> tag without requiring an additional fetch request. The script also updates README statistics, syncs badge counts, generates a sitemap, and creates an llms.txt file for AI agent consumption.

Generated Data Structure

The resulting site/data.js contains three top-level constants that power the website's navigation and search:

const PHASES = [
  {
    "id": 0,
    "name": "Setup & Tooling",
    "status": "complete",
    "desc": "Get your dev environment ready.",
    "lessons": [
      {
        "name": "Dev Environment",
        "status": "complete",
        "type": "Build",
        "lang": "Python, TypeScript, Rust",
        "url": "https://github.com/rohitg00/ai-engineering-from-scratch/tree/main/phases/00-setup-and-tooling/01-dev-environment/",
        "summary": "Set up a reproducible Python/TS/Rust dev stack.",
        "keywords": "Virtualenv · Cargo · npm"
      }
    ]
  }
];

const GLOSSARY = [
  {
    "term": "Attention",
    "says": "Focus on the most relevant parts of the input.",
    "means": "A weighted sum of value vectors where weights come from queries and keys."
  }
];

const ARTIFACTS = [
  {
    "kind": "skill",
    "name": "Prompt‑Engineer",
    "description": "A reusable prompt template for instruction‑following models.",
    "tags": ["prompt", "template"],
    "phase": 7,
    "lesson": 3,
    "lessonPath": "phases/07-foundations/03-prompt-engineer",
    "file": "phases/07-foundations/03-prompt-engineer/outputs/skill-prompt-engineer.md"
  }
];

The literal JavaScript format ensures immediate availability upon page load, supporting the site's client-side rendering without asynchronous data fetching.

Automating the Build Process

The repository automates site/build.js execution through a GitHub Actions workflow defined in .github/workflows/curriculum.yml. The workflow triggers on every push, ensuring site/data.js remains synchronized with the latest markdown curriculum changes.

For local development, regenerate the data file manually by running:

node site/build.js

This command updates site/data.js after editing any curriculum markdown, including README updates, lesson content changes, or glossary additions.

Summary

• site/build.js serves as the single source of truth for curriculum data generation, parsing README.md, ROADMAP.md, and glossary/terms.md to create a unified data bundle.
• The script produces three JavaScript constants—PHASES, GLOSSARY, and ARTIFACTS—that contain complete curriculum metadata, term definitions, and reusable code components.
• Lesson enrichment occurs through extractLessonMeta(), which pulls summaries from blockquotes and keywords from H3 headings in each lesson's docs/en.md file.
• Artefact discovery automatically indexes skills, prompts, and missions from phases/*/*/outputs/ directories, making practical examples searchable.
• The generated site/data.js uses literal JavaScript syntax for immediate browser consumption, eliminating the need for runtime JSON parsing.
• The build process runs automatically via GitHub Actions on every push, or manually via node site/build.js for local development.

Frequently Asked Questions

How does site/build.js determine the completion status of each lesson?

The script parses ROADMAP.md using the parseRoadmap() function to extract emoji-based status indicators. It maps ✅ (complete), 🚧 (in-progress), and ⬚ (planned) emojis found in phase headers and lesson table rows to standardized status strings (complete, in-progress, planned). These statuses are then attached to the corresponding lesson objects in the PHASES array during the README parsing phase.

What file paths does the build script scan for lesson documentation?

The script looks for lesson-specific documentation at phases/{phase}/{lesson}/docs/en.md relative to the repository root. The extractLessonMeta() function constructs this path by combining the relative lesson path (extracted from the README's GitHub URLs) with the fixed docs/en.md suffix. It then reads the first blockquote for the summary and collects all H3 headings as keywords.

Can I run the build script locally without triggering the GitHub Actions workflow?

Yes. Run node site/build.js from the repository root to manually regenerate site/data.js. This executes the same parsing logic used in the CI pipeline, updating the data file with any local changes you've made to README.md, ROADMAP.md, lesson documentation, or glossary terms. The script will overwrite site/data.js with fresh serialized data containing the current timestamp.

What is the difference between the PHASES and ARTIFACTS constants in data.js?

The PHASES constant contains the hierarchical curriculum structure—phases and lessons with their metadata, URLs, and completion statuses—essentially representing the learning path. The ARTIFACTS constant indexes reusable components discovered in lesson outputs/ folders, such as prompt templates, agent configurations, and skill definitions. While PHASES organizes content for human navigation, ARTIFACTS provides machine-readable references to practical code components that developers can reuse in their own AI engineering projects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}