# Book Generation Pipeline: How AI Engineering from Scratch Renders Markdown to EPUB and PDF > Learn the book generation pipeline in ai-engineering-from-scratch. Discover how CI processes Markdown lessons into EPUB and PDF formats using Python and Pandoc. - Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch) - Tags: architecture - Published: 2026-08-29 --- **The book generation pipeline in `rohitg00/ai-engineering-from-scratch` automatically assembles distributed lesson Markdown into unified manuscripts using [`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py), then renders them into EPUB and PDF formats via Pandoc within GitHub Actions CI.** The `rohitg00/ai-engineering-from-scratch` repository employs a sophisticated **book generation pipeline** to convert discrete curriculum files into professional e-book artifacts. This automated workflow aggregates content from multiple learning phases, performs source-to-source transformations on Markdown files, and produces distributable EPUB and PDF outputs through a continuous integration process. ## Configuration and Volume Definitions ### Volume Mapping in [`book/volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/volumes.json) The pipeline begins by reading **[`book/volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/volumes.json)**, which defines each book volume’s metadata including its number, slug, title, subtitle, and the `lesson_dirs` array that maps phase directories to volume contents. This configuration determines which lessons from the `phases/` directory tree are concatenated into a single book manuscript. ### Source File Resolution For each volume, the build script locates lesson content at **`phases/*/docs/en.md`** by default. When building translated editions via the `--lang` flag, the pipeline sources files from **`i18n//…/docs/*.md`** instead, enabling localized book generation without modifying the core curriculum structure. ## Core Assembly Logic in [`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py) The Python script **[`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py)** implements three primary functions that orchestrate the pipeline: `transform_lesson()`, `assemble()`, and `render()`. ### Lesson Transformation The `transform_lesson()` function processes individual Markdown files to prepare them for static book formats. It rewrites relative image links to ensure they resolve correctly in the assembled document, expands special fenced code blocks (such as ` ```figure` and ` ```mermaid`) into interactive placeholders, and injects *"Continue online"* boxes that hyperlink back to the live web edition and runnable code repositories. These transformations ensure that dynamic web content degrades gracefully into static book formats while maintaining reference integrity. ### Manuscript Assembly The `assemble()` function concatenates all transformed lessons for a given volume into a single Markdown file written to **`book/_build/.md`**. During this phase, the script also generates a YAML metadata file containing pandoc-specific front matter (title, author, language settings) required for rendering. The assembly process preserves the pedagogical order defined in [`volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/volumes.json) and aggregates word counts for progress reporting. ## Rendering EPUB and PDF Formats ### EPUB Generation via `render()` The `render()` function (lines 29-34 of the script) executes pandoc to convert the assembled Markdown into EPUB format. It applies the stylesheet **[`book/epub.css`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/epub.css)** and outputs the final artifact to **`dist/book/aiefs-vol-.epub`**. The EPUB renderer respects the YAML metadata generated during assembly, ensuring proper title pages and table of contents generation. ### PDF Compilation with XeLaTeX When invoked with the `--pdf` flag, the pipeline triggers an additional rendering pass using pandoc’s `xelatex` PDF engine. This process utilizes **`book/titlepage.tex`** as a LaTeX template for professional title page formatting and calls the `pick_font()` function to select appropriate typefaces for the target language. The PDF pipeline writes outputs to **`dist/book/aiefs-vol-.pdf`**. **Note:** The PDF generation step is automatically skipped for RTL (right-to-left) languages because the current LaTeX theme does not support bidirectional text rendering. ## CI/CD Automation with GitHub Actions ### Workflow Orchestration in [`.github/workflows/build-book.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/build-book.yml) The continuous integration workflow defined in **[`.github/workflows/build-book.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/build-book.yml)** automates the entire book generation pipeline on every push and tag event. According to the repository’s CI configuration, the workflow performs the following steps: - Installs system dependencies including `pandoc`, `mmdc` (Mermaid CLI for diagram rendering), and font utilities required for PDF generation. - Executes the build command: `python3 scripts/build_book.py --pdf` to generate both EPUB and PDF artifacts. - Publishes the resulting `*.epub` and `*.pdf` files to GitHub Releases using the `gh release upload` command, making the books immediately available via the repository’s Releases page. ## Running the Pipeline Locally You can execute the book generation pipeline locally to preview outputs or build custom language editions using the following commands: ```bash # Build every volume, generating both EPUB and PDF outputs: python3 scripts/build_book.py --pdf # Build only the "language" volume, English edition, without PDF: python3 scripts/build_book.py --volume language # Build a translated edition (e.g., French) and produce EPUB only: python3 scripts/build_book.py --lang fr ``` The script provides progress output indicating chapter counts, word totals, and final file sizes: ```text vol 2 deep-learning: 42 chapters, 123,456 words -> book/_build/deep-learning.md dist/book/aiefs-vol2-deep-learning.epub (4 250 KB) dist/book/aiefs-vol2-deep-learning.pdf (5 800 KB) ``` ## Key File Reference - **[`book/volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/volumes.json)**: Defines volume metadata and phase-to-book mappings. - **[`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py)**: Core Python logic for assembly, transformation, and rendering. - **[`book/epub.css`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/epub.css)**: Cascading stylesheet applied to generated EPUB files. - **`book/titlepage.tex`**: LaTeX template used for PDF title page generation. - **[`.github/workflows/build-book.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/build-book.yml)**: GitHub Actions workflow that automates builds and releases. - **`i18n//…/docs/*.md`**: Localized lesson source files used when the `--lang` parameter is specified. ## Summary - The **book generation pipeline** drives the `ai-engineering-from-scratch` project’s ability to publish curriculum as static e-books. - **[`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py)** orchestrates the process through `transform_lesson()`, `assemble()`, and `render()` functions, handling Markdown from `phases/` or `i18n/` directories. - **Pandoc** converts assembled manuscripts into EPUB (using [`book/epub.css`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/epub.css)) and PDF (using `xelatex` and `book/titlepage.tex`), with PDF generation skipping RTL languages. - The **GitHub Actions** workflow ([`.github/workflows/build-book.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/build-book.yml)) automates builds on every push, installing `pandoc` and `mmdc` before uploading artifacts to Releases. ## Frequently Asked Questions ### What configuration file determines which lessons are included in each book volume? The **[`book/volumes.json`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/book/volumes.json)** file defines the mapping between learning phases and book volumes. It specifies each volume’s slug, title, and the ordered list of phase directories (`lesson_dirs`) that the `assemble()` function will process into a single manuscript. ### How does the book generation pipeline handle translations and internationalization? When you pass the `--lang ` flag to [`scripts/build_book.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/scripts/build_book.py), the pipeline sources lesson content from **`i18n//…/docs/*.md`** instead of the default `phases/*/docs/en.md` locations. The `pick_font()` function in the build script then selects appropriate fonts for the specified language during PDF generation, though RTL languages are currently excluded from PDF output due to LaTeX limitations. ### Why are Mermaid diagrams and figure blocks transformed during the build process? The `transform_lesson()` function converts dynamic web elements—such as ` ```mermaid` and ` ```figure` fenced blocks—into static interactive placeholders suitable for e-readers. This transformation ensures that complex diagrams and interactive content are represented appropriately in the static EPUB and PDF formats while preserving references to the online runnable versions. ### What triggers the automated book builds and where are the artifacts published? The **[`.github/workflows/build-book.yml`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/.github/workflows/build-book.yml)** GitHub Actions workflow triggers on every git push and tag event. After installing dependencies like `pandoc` and `mmdc`, it runs the build script with `--pdf` enabled and publishes the resulting `aiefs-vol-.epub` and `.pdf` files directly to the repository’s GitHub Releases page using `gh release upload`.