How the Six-Volume Book Series Is Generated from Lesson Sources in AI Engineering from Scratch

The six-volume AI Engineering from Scratch book series is generated by an automated Python pipeline that maps course phases to volumes, transforms individual lesson markdown files with interactive web links, and compiles them into EPUB and PDF formats using Pandoc.

The rohitg00/ai-engineering-from-scratch repository contains a sophisticated build system that converts modular lesson sources into a cohesive six-volume book series. This automated pipeline bridges the gap between living online course content and static publication formats. Understanding how the six-volume book series is generated from lesson sources reveals the architecture behind maintaining synchronized web and print educational materials.

Volume Definition and Phase Mapping

The pipeline begins with a static JSON configuration that establishes the structure of the six-volume book series. The file book/volumes.json maps each volume to specific course phases, defining the organization of the entire curriculum.

{
  "volumes": [
    { 
      "slug": "foundations", 
      "number": 1, 
      "title": "Foundations", 
      "phases": ["00-setup-and-tooling", "01-math-foundations", "02-ml-fundamentals"] 
    },
    {
      "slug": "production",
      "number": 6,
      "title": "Production",
      "phases": ["17-infrastructure-and-production", "18-ethics-safety-alignment", "19-capstone-projects"]
    }
  ]
}

This configuration allows the build script to iterate through specific phases and aggregate the correct lesson sources for each volume. The phase-based organization ensures that the linear progression of the course maintains logical continuity across the six separate books.

Lesson Discovery and Directory Scanning

For each phase defined in the JSON mapping, the build system scans the filesystem to locate actual lesson content. The lesson_dirs() function in scripts/build_book.py (lines 45-53) searches the phases/<phase>/ directory structure for folders containing a docs/en.md file.

This discovery mechanism treats the repository's folder structure as the single source of truth. By scanning for docs/en.md within each phase directory, the pipeline automatically includes new lessons as they are added to the course without requiring manual updates to the build configuration.

Transforming Lesson Content for Print

Once discovered, each lesson markdown undergoes transformation via the transform_lesson() function in scripts/build_book.py (lines 9-80). This critical step adapts web-native content for static book formats while preserving references to interactive elements.

URL Generation and Web Integration

The urls_for() function generates three essential URLs for every lesson:

def urls_for(phase, lesson):
    rel = f"phases/{phase}/{lesson}"
    return {
        "web": f"{SITE}/lesson?path={rel}",
        "code": f"{REPO}/tree/main/{rel}/code",
        "repo": f"{REPO}/tree/main/{rel}",
    }

These URLs enable the "continue online" pattern—every printed chapter directs readers back to the live web edition for interactive features, updated code, and browser-graded quizzes.

Markdown Transformation Pipeline

The transform_lesson() function performs several specific content conversions:

• Figure blocks: Converts ```figure fenced blocks into boxed notices linking to interactive figures on the web edition

• Mermaid diagrams: Renders ```mermaid blocks into SVG using the Mermaid CLI when available, or falls back to web-edition pointers

• Section replacement: Substitutes "## Ship It" and "## Exercises" sections with "continue-online" boxes pointing to starter code and solutions

• Asset path rewriting: Adjusts image paths so Pandoc can properly embed assets during compilation

• Continuation boxes: Appends a standardized box containing web links, code repositories, and quiz locations to every lesson

The continue-online box implementation ensures readers always have access to the most current version of the material:

def continue_box(u, has_quiz):
    lines = [
        "**Continue online.** The living edition of this chapter has more than the page can hold:",
        "",
        f"- Animated, interactive figures and the web text: <{u['web']}>",
        f"- Runnable code for every step: <{u['code']}>",
    ]
    if has_quiz:
        lines.append(f"- The chapter quiz, graded in the browser: <{u['web']}>")
    lines += [
        "",
        "The repository moves faster than any printing. When the book and the repo disagree, trust the repo.",
    ]
    return fenced_div("continue-online", *lines)

Volume Assembly and Rendering

The assemble() function in scripts/build_book.py (lines 89-106) constructs the final markdown for each volume by concatenating transformed lessons. This process includes:

1. Inserting "How to use this volume" front-matter via how_to_use()
2. Adding "Part I/II/..." headings for every phase transition
3. Counting chapters and words for metadata tracking

After assembly, the render() function (lines 22-44) writes a YAML metadata header and invokes Pandoc to generate the EPUB format. When the --pdf flag is provided and the language is left-to-right, the pipeline also produces PDF output using XeLaTeX with DejaVu fonts.

You can execute the build process using the command-line interface:

# Build every volume as EPUB (default)

python3 scripts/build_book.py

# Build only the "language" volume

python3 scripts/build_book.py --volume language

# Build EPUB + PDF for all volumes (requires xelatex and DejaVu fonts)

python3 scripts/build_book.py --pdf

Continuous Integration and Automated Publishing

A GitHub Actions workflow defined in .github/workflows/build-book.yml automates the entire pipeline. The CI triggers on every push to the phases/ directory tree, ensuring the six-volume book series remains synchronized with the latest lesson sources.

The workflow publishes generated EPUBs to dist/book/ and creates PDF artifacts for release tags. This automation maintains the principle that while the book serves as a snapshot of the course, the repository remains the authoritative source.

Summary

• JSON configuration in book/volumes.json maps specific course phases to each of the six volumes, establishing the book structure.
• Automated discovery via lesson_dirs() scans phase directories for docs/en.md files, dynamically including new lessons.
• Content transformation through transform_lesson() converts web-native markdown to print-ready formats while embedding "continue online" references.
• Pandoc compilation generates EPUB output with optional PDF rendering via XeLaTeX for supported languages.
• GitHub Actions provides continuous integration, automatically rebuilding the book series when lesson sources change.

Frequently Asked Questions

What file formats does the six-volume book series generate?

The pipeline produces EPUB as the default format for all volumes. When the --pdf flag is specified and the language direction is left-to-right, the system also generates PDF files using XeLaTeX. Final artifacts are stored in dist/book/ with filenames following the pattern aiefs-volX-<slug>.epub.

How does the build system handle interactive figures and Mermaid diagrams?

The transform_lesson() function processes ```figure blocks by converting them into boxed notices that link to the interactive web edition. For ```mermaid blocks, the pipeline attempts SVG rendering via the Mermaid CLI; if unavailable, it falls back to web-edition pointers. This ensures static books reference dynamic content without breaking the reading experience.

Can I build individual volumes instead of the entire series?

Yes. The scripts/build_book.py script accepts a --volume argument that accepts a volume slug (such as language or foundations). Running python3 scripts/build_book.py --volume language will process only the phases mapped to that specific volume in book/volumes.json, significantly reducing build time during development.

Where does the pipeline store intermediate and final files?

Intermediate markdown assemblies are written to book/_build/<slug>.md during the assemble() phase. Final EPUB artifacts are published to dist/book/aiefs-volX-<slug>.epub, with PDF variants stored in the same directory when generated. The .github/workflows/build-book.yml CI configuration manages these paths during automated builds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}