How the AI Engineering from Scratch Static Site is Built: Inside the site/data.js Pipeline

The ai-engineering-from-scratch static site is generated by a Node.js build script (site/build.js) that parses Markdown curriculum files and exports three constants—PHASES, GLOSSARY, and ARTIFACTS—into site/data.js, which serves as the runtime data layer for the client-side UI.

The public website aiengineeringfromscratch.com operates as a pure static site with zero server-side rendering. According to the rohitg00/ai-engineering-from-scratch repository, the build process aggregates curriculum metadata, lesson statuses, and reusable artifacts into a single JavaScript module consumed by the frontend HTML pages.

Build Architecture and CI Integration

The static site generation is fully automated through GitHub Actions. On every push to the repository, the workflow defined in .github/workflows/curriculum.yml triggers the build command:

node site/build.js

This script reads the source Markdown files—including README.md, ROADMAP.md, and glossary/terms.md—and writes the generated data to site/data.js. Because the generated file is listed in .gitignore, the repository stores only the source of truth, while CI produces the runtime artifacts fresh on every deployment.

The Build Pipeline in site/build.js

The site/build.js script executes an eight-step pipeline to transform raw curriculum Markdown into structured JavaScript data. Each step uses specific helper functions to extract and normalize content.

Parsing Source Files and Lesson Status

The build begins by reading the primary source files using fs.readFileSync (lines 4‑8 in site/build.js). The parseRoadmap() function (lines 30‑61) processes ROADMAP.md to detect status emojis—✅ for complete, 🚧 for in-progress, and ⬚ for planned—and builds a lookup table mapping phases to lesson completion states.

Extracting Phase and Lesson Metadata

The parseReadme() function (lines 63‑131) scans README.md for Markdown tables containing the master curriculum list. It handles both the compact Phase‑0 table and the <details> blocks for subsequent phases, extracting lesson names, types, programming languages, and GitHub URLs. For each lesson discovered, extractLessonMeta() (lines 33‑48) reads the corresponding docs/en.md file to pull the first blockquote (used as a summary) and all ### headings (used as keywords), enriching the lesson objects with these metadata fields.

Discovering Reusable Artifacts

The discoverArtifacts() function (lines 39‑98) crawls the directory structure under phases/*/*/outputs/*.md to locate skill, prompt, and agent definitions. It reads YAML front-matter from each Markdown file (delimited by ---) to extract name, description, tags, and other metadata, compiling these into the ARTIFACTS array. This function also detects optional mission.md files to include mission-specific context.

Assembling the Final Payload

The build() function (lines 100‑162) orchestrates the final assembly, combining the phase objects, glossary terms, and artifacts into three arrays. It serializes these into a JavaScript template string and writes the output to site/data.js. Following this, helper functions like syncCounts(), writeSitemap(), and writeLlms() (lines 162‑225) generate auxiliary files including stats.json for badge synchronization, sitemap.xml for SEO, and llms.txt for LLM-friendly indexing.

The Critical Role of site/data.js

site/data.js serves as the bridge between the author-centric Markdown source layout and the viewer-centric static site. The file exports three top-level constants:

const PHASES = [...];   // phases → lessons → metadata
const GLOSSARY = [...]; // term definitions
const ARTIFACTS = [...]; // skills / prompts / agents / missions

This architecture provides several key advantages:

• Single Source of Truth: The frontend never parses Markdown directly. All lesson data, glossary entries, and artifact descriptions are pre-computed, eliminating duplicate parsing logic in the browser.
• Fast Client-Side Rendering: As a plain JavaScript module cached like any static asset, data.js enables instant UI hydration without backend queries.
• Consistent URL Mapping: Each lesson receives a numeric id derived from directory names, enabling the frontend to construct clean URLs like /lesson.html?path=phases/03-deep-learning-core/01-the-perceptron.
• Extensible Schema: Adding new content types requires only updating discoverArtifacts() in the build script; the frontend automatically receives new data through the existing ARTIFACTS export.

Running the Build Locally

To generate the static data locally for development or debugging:

# From the repository root

node site/build.js

Upon completion, the script outputs site/data.js alongside status logs:

📖 Reading source files...
🔍 Parsing ROADMAP.md...
🔍 Parsing README.md...
✅ Generated site/data.js

Consuming Generated Data in the Browser

The frontend HTML pages import the generated module directly to render dynamic content:

<script type="module">
  import { PHASES, GLOSSARY, ARTIFACTS } from './data.js';

  // Access Phase 3 lessons
  const phase3 = PHASES.find(p => p.id === 3);
  console.log('Phase 3 lessons:', phase3.lessons.map(l => l.name));

  // Query the glossary
  console.log('First term:', GLOSSARY[0]);

  // Filter for reusable prompts
  const prompts = ARTIFACTS.filter(a => a.kind === 'prompt');
  console.log(`Found ${prompts.length} reusable prompts`);
</script>

When constructing lesson URLs, the site uses the path extracted from the GitHub URL stored in each lesson object, typically matching the pattern phases/XX-topic/YY-lesson-name.

Summary

• The ai-engineering-from-scratch static site is built by site/build.js, a Node.js script that parses README.md, ROADMAP.md, and artifact files to generate site/data.js.
• The build pipeline uses functions like parseRoadmap(), parseReadme(), and discoverArtifacts() to extract metadata, lesson statuses, and reusable content.
• site/data.js exports PHASES, GLOSSARY, and ARTIFACTS as the single runtime data source for the client-side UI.
• The process runs automatically via .github/workflows/curriculum.yml on every push, ensuring the site stays synchronized with the Markdown source.
• Generated files including data.js, sitemap.xml, and llms.txt are git-ignored and produced fresh during CI/CD.

Frequently Asked Questions

How does the build script determine if a lesson is complete?

The script uses the parseRoadmap() function (lines 30‑61) to scan ROADMAP.md for Unicode emojis. A ✅ indicates a complete lesson, 🚧 marks work in progress, and ⬚ denotes planned content. These statuses are mapped to a lookup table and merged with the lesson metadata extracted from README.md.

Why is site/data.js not checked into the repository?

site/data.js is listed in .gitignore to prevent stale data from being committed. Because the file is regenerated on every push by the GitHub Actions workflow, storing it in version control would create unnecessary merge conflicts and potentially serve outdated curriculum data to users.

How are skills, prompts, and agents discovered during the build?

The discoverArtifacts() function (lines 39‑98) recursively scans phases/*/*/outputs/*.md files. It reads the YAML front-matter between --- delimiters to extract metadata fields like name, description, and tags, then categorizes each file into the ARTIFACTS array based on its type and location.

Can I run the static site build without GitHub Actions?

Yes. Running node site/build.js from the repository root executes the full pipeline locally using Node.js. This generates site/data.js, sitemap.xml, llms.txt, and other auxiliary files, allowing you to preview changes or debug the curriculum data before pushing to the remote repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}