How site/build.js Generates site/data.js from README and ROADMAP: The Build Pipeline Explained

The site/build.js script transforms human-readable curriculum documentation into a structured JavaScript module by parsing README.md and ROADMAP.md, enriching the data with URLs and SEO metadata, and serializing the results to site/data.js for the static site.

The rohitg00/ai-engineering-from-scratch repository maintains its curriculum as markdown documentation, using site/build.js as the central orchestration tool that generates the website's data layer. This Node.js script ensures the single source of truth remains in the readable documentation files while producing the structured data required by the frontend.

Loading the Curriculum Source Files

At lines 15-19 of site/build.js, the script resolves absolute paths for the repository root and defines constants for the three primary documentation sources:

• REPO_ROOT: The repository base directory
• README_PATH: The master curriculum table (README.md)
• ROADMAP_PATH: The completion status tracker (ROADMAP.md)
• GLOSSARY_PATH: The searchable terms and definitions (glossary/terms.md)

Parsing the ROADMAP Status Map

The parseRoadmap() function (lines 101-133) processes ROADMAP.md to extract completion states. It iterates line-by-line to identify phase headings marked with status emojis—✅ (complete), 🚧 (in progress), or ⬚ (planned)—and captures the status of each lesson row.

The function returns a nested map structure:

{
  "Phase 1": {
    phaseStatus: "complete",
    lessons: {
      "Lesson Title": "complete",
      "Another Lesson": "in-progress"
    }
  }
}

Extracting Lesson Metadata from README

The parseReadme() function (core logic at lines 138-286) handles the heavy lifting of parsing the master table in README.md. It performs two critical tasks:

Phase Header Detection: The script recognizes phase headers across multiple markup variants, establishing the current phase context for subsequent lesson rows.

Lesson Row Processing (lines 208-286): For each table row, the script extracts:

• Lesson name and optional link
• Type classification (Build or Learn)
• Language identifier (plain text or emoji)
• URL construction using GITHUB_BASE when links are present

Crucially, the function cross-references the roadmap map generated earlier, attaching the appropriate status to each lesson and defaulting to "planned" when no match exists in the roadmap data.

Enriching Data with Learning Paths and SEO

Beyond the core curriculum, the pipeline gathers supplemental information and search metadata.

Learning Paths: The parseLearningPaths() function (lines 307-424) reads JSON files from the learning-paths/ directory to create custom ordered overlays on the standard curriculum structure.

SEO Manifest Generation: The buildSeoManifests() function (lines 190-247) creates search-optimized metadata for each lesson. It reads individual lesson documentation (such as docs/en.md), extracts summaries and keywords, and generates SEO-friendly titles and descriptions. The validation logic (lines 291-423) enforces strict technical SEO limits:

• Title ≤ 60 characters
• Description ≤ 160 characters
• Excerpt ≤ 220 words

The function also validates canonical URL uniqueness to prevent duplicate content issues.

Serializing the Output Artifacts

The final stage writes multiple artifacts to disk. The writeFigureManifest() function (lines 624-633) generates figure-manifest.js for custom diagram components, while writeSeoArtifacts() (lines 818-829) produces lesson-seo.json and certification-seo.json.

Finally, the script constructs the main output file. At approximately lines 1240-1255, it calls fs.writeFileSync(OUTPUT_PATH, ...) where OUTPUT_PATH is defined at line 19 as site/data.js. This writes a JavaScript module exporting the complete curriculum object:

export default {
  phases: [...],
  lessons: [...],
  glossary: [...],
  seoManifest: {...}
};

Regenerating the Site Data

To execute the build pipeline locally, run the script from the repository root:

node site/build.js

The script outputs diagnostic logs indicating the files written:

wrote figure-manifest.js (123 routed figures)
wrote lesson-seo.json (340 lessons)
wrote certification-seo.json (12 tracks)

Consuming the generated module in the frontend requires a standard ES module import:

<script type="module">
  import data from './data.js';
  console.log(data.phases);        // Array of phase objects
  console.log(data.lessonManifest); // SEO-ready lesson metadata
</script>

Summary

• site/build.js operates as the single-source build script for the rohitg00/ai-engineering-from-scratch curriculum site, processing markdown into structured data.
• The parsing phase extracts structured data from README.md (lesson metadata) and ROADMAP.md (completion status with ✅/🚧/⬚ emojis).
• Cross-referencing occurs during parseReadme(), where roadmap status values are mapped to individual lessons, defaulting to "planned" when unspecified.
• Validation logic in buildSeoManifests() enforces technical SEO constraints including character limits for titles (≤60) and descriptions (≤160).
• The script generates multiple artifacts: site/data.js (main data module), lesson-seo.json, certification-seo.json, and figure-manifest.js.
• Running node site/build.js regenerates the entire data layer from markdown source files, maintaining synchronization between documentation and the deployed site.

Frequently Asked Questions

What happens if a lesson exists in README.md but not in ROADMAP.md?

The parseReadme() function defaults the lesson status to "planned" when no matching entry exists in the roadmap map. This ensures the site displays all documented curriculum items even if their implementation status hasn't been explicitly tracked in the roadmap file.

How does the script handle different markdown formatting for phase headers?

The parseReadme() function at lines 138-210 implements flexible header detection that recognizes multiple markup variants for phase declarations. This robust parsing accommodates inconsistent formatting across the long-form markdown document while correctly grouping lessons under their respective phases.

Where does the SEO validation logic enforce character limits?

The buildSeoManifests() function contains validation checks at lines 291-423 that verify SEO metadata adheres to technical constraints: titles must not exceed 60 characters, descriptions must stay within 160 characters, and excerpts are limited to 220 words. The script throws descriptive errors if content violates these limits.

Can this build process be automated in CI/CD pipelines?

Yes, site/build.js is a standard Node.js script with no interactive dependencies. You can execute node site/build.js within any CI/CD environment that provides Node.js, allowing automated regeneration of site/data.js and SEO artifacts whenever README.md or ROADMAP.md changes are merged into the main branch.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}