Book Generation Pipeline: How AI Engineering from Scratch Renders Markdown to EPUB and PDF

The book generation pipeline in rohitg00/ai-engineering-from-scratch automatically assembles distributed lesson Markdown into unified manuscripts using scripts/build_book.py, then renders them into EPUB and PDF formats via Pandoc within GitHub Actions CI.

The rohitg00/ai-engineering-from-scratch repository employs a sophisticated book generation pipeline to convert discrete curriculum files into professional e-book artifacts. This automated workflow aggregates content from multiple learning phases, performs source-to-source transformations on Markdown files, and produces distributable EPUB and PDF outputs through a continuous integration process.

Configuration and Volume Definitions

Volume Mapping in book/volumes.json

The pipeline begins by reading book/volumes.json, which defines each book volume’s metadata including its number, slug, title, subtitle, and the lesson_dirs array that maps phase directories to volume contents. This configuration determines which lessons from the phases/ directory tree are concatenated into a single book manuscript.

Source File Resolution

For each volume, the build script locates lesson content at phases/*/docs/en.md by default. When building translated editions via the --lang flag, the pipeline sources files from i18n/<lang>/…/docs/*.md instead, enabling localized book generation without modifying the core curriculum structure.

Core Assembly Logic in scripts/build_book.py

The Python script scripts/build_book.py implements three primary functions that orchestrate the pipeline: transform_lesson(), assemble(), and render().

Lesson Transformation

The transform_lesson() function processes individual Markdown files to prepare them for static book formats. It rewrites relative image links to ensure they resolve correctly in the assembled document, expands special fenced code blocks (such as ```figure and ```mermaid) into interactive placeholders, and injects "Continue online" boxes that hyperlink back to the live web edition and runnable code repositories. These transformations ensure that dynamic web content degrades gracefully into static book formats while maintaining reference integrity.

Manuscript Assembly

The assemble() function concatenates all transformed lessons for a given volume into a single Markdown file written to book/_build/<slug>.md. During this phase, the script also generates a YAML metadata file containing pandoc-specific front matter (title, author, language settings) required for rendering. The assembly process preserves the pedagogical order defined in volumes.json and aggregates word counts for progress reporting.

Rendering EPUB and PDF Formats

EPUB Generation via render()

The render() function (lines 29-34 of the script) executes pandoc to convert the assembled Markdown into EPUB format. It applies the stylesheet book/epub.css and outputs the final artifact to dist/book/aiefs-vol<N>-<slug>.epub. The EPUB renderer respects the YAML metadata generated during assembly, ensuring proper title pages and table of contents generation.

PDF Compilation with XeLaTeX

When invoked with the --pdf flag, the pipeline triggers an additional rendering pass using pandoc’s xelatex PDF engine. This process utilizes book/titlepage.tex as a LaTeX template for professional title page formatting and calls the pick_font() function to select appropriate typefaces for the target language. The PDF pipeline writes outputs to dist/book/aiefs-vol<N>-<slug>.pdf.

Note: The PDF generation step is automatically skipped for RTL (right-to-left) languages because the current LaTeX theme does not support bidirectional text rendering.

CI/CD Automation with GitHub Actions

Workflow Orchestration in .github/workflows/build-book.yml

The continuous integration workflow defined in .github/workflows/build-book.yml automates the entire book generation pipeline on every push and tag event. According to the repository’s CI configuration, the workflow performs the following steps:

• Installs system dependencies including pandoc, mmdc (Mermaid CLI for diagram rendering), and font utilities required for PDF generation.
• Executes the build command: python3 scripts/build_book.py --pdf to generate both EPUB and PDF artifacts.
• Publishes the resulting *.epub and *.pdf files to GitHub Releases using the gh release upload command, making the books immediately available via the repository’s Releases page.

Running the Pipeline Locally

You can execute the book generation pipeline locally to preview outputs or build custom language editions using the following commands:

# Build every volume, generating both EPUB and PDF outputs:

python3 scripts/build_book.py --pdf

# Build only the "language" volume, English edition, without PDF:

python3 scripts/build_book.py --volume language

# Build a translated edition (e.g., French) and produce EPUB only:

python3 scripts/build_book.py --lang fr

The script provides progress output indicating chapter counts, word totals, and final file sizes:

vol 2 deep-learning: 42 chapters, 123,456 words -> book/_build/deep-learning.md
  dist/book/aiefs-vol2-deep-learning.epub (4 250 KB)
  dist/book/aiefs-vol2-deep-learning.pdf (5 800 KB)

Key File Reference

• book/volumes.json: Defines volume metadata and phase-to-book mappings.
• scripts/build_book.py: Core Python logic for assembly, transformation, and rendering.
• book/epub.css: Cascading stylesheet applied to generated EPUB files.
• book/titlepage.tex: LaTeX template used for PDF title page generation.
• .github/workflows/build-book.yml: GitHub Actions workflow that automates builds and releases.
• i18n/<lang>/…/docs/*.md: Localized lesson source files used when the --lang parameter is specified.

Summary

• The book generation pipeline drives the ai-engineering-from-scratch project’s ability to publish curriculum as static e-books.
• scripts/build_book.py orchestrates the process through transform_lesson(), assemble(), and render() functions, handling Markdown from phases/ or i18n/ directories.
• Pandoc converts assembled manuscripts into EPUB (using book/epub.css) and PDF (using xelatex and book/titlepage.tex), with PDF generation skipping RTL languages.
• The GitHub Actions workflow (.github/workflows/build-book.yml) automates builds on every push, installing pandoc and mmdc before uploading artifacts to Releases.

Frequently Asked Questions

What configuration file determines which lessons are included in each book volume?

The book/volumes.json file defines the mapping between learning phases and book volumes. It specifies each volume’s slug, title, and the ordered list of phase directories (lesson_dirs) that the assemble() function will process into a single manuscript.

How does the book generation pipeline handle translations and internationalization?

When you pass the --lang <code> flag to scripts/build_book.py, the pipeline sources lesson content from i18n/<lang>/…/docs/*.md instead of the default phases/*/docs/en.md locations. The pick_font() function in the build script then selects appropriate fonts for the specified language during PDF generation, though RTL languages are currently excluded from PDF output due to LaTeX limitations.

Why are Mermaid diagrams and figure blocks transformed during the build process?

The transform_lesson() function converts dynamic web elements—such as ```mermaid and ```figure fenced blocks—into static interactive placeholders suitable for e-readers. This transformation ensures that complex diagrams and interactive content are represented appropriately in the static EPUB and PDF formats while preserving references to the online runnable versions.

What triggers the automated book builds and where are the artifacts published?

The .github/workflows/build-book.yml GitHub Actions workflow triggers on every git push and tag event. After installing dependencies like pandoc and mmdc, it runs the build script with --pdf enabled and publishes the resulting aiefs-vol<N>-<slug>.epub and .pdf files directly to the repository’s GitHub Releases page using gh release upload.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}