# How Book-to-Skill Generates Per-Chapter Summaries: A Technical Deep Dive

> Discover how Book-to-Skill generates per-chapter summaries with its technical pipeline. Learn about text extraction, content splitting, and LLM abstract validation for structured summaries. Explore the virgiliojr94/book-to-skil...

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-29

---

**Book-to-Skill generates per-chapter summaries through a three-stage pipeline that extracts raw text from PDFs or EPUBs, splits content using deterministic heuristics in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), and validates LLM-generated abstracts via [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py) to ensure every chapter has a structured summary in [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json).**

The virgiliojr94/book-to-skill open-source project transforms static books into actionable skill artifacts by creating structured per-chapter summaries for each section of a document. This process combines deterministic text processing with LLM-powered abstraction to convert unstructured book formats—PDF, EPUB, DOCX, and HTML—into validated JSON metadata. Understanding how Book-to-Skill generates these per-chapter summaries reveals a tightly-coupled pipeline of extraction, detection, and validation.

## The Three-Stage Pipeline for Per-Chapter Summaries

Book-to-Skill processes books through three tightly-coupled stages that convert raw binary documents into structured, chapter-level metadata.

### Stage 1: Text Extraction

The pipeline begins when [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) invokes the CLI entry point in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py). This component calls the appropriate parser for the input format—whether PDF, EPUB, HTML, or DOCX—and writes the extracted plain text to a temporary working directory.

According to the source code in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), the system defines `OUTPUT_TEXT` as the destination for the full book content, typically located at `book_skill_work-<pid>/full_text.txt`. This file contains the complete, unprocessed text of the source document, serving as the input for the subsequent chapter detection phase.

### Stage 2: Chapter Detection with Heuristics

Once extraction completes, the system parses the monolithic text file into discrete chapters using deterministic heuristics implemented in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py). The core function `split_chapters(text: str) → list[tuple[int | None, str, str]]` (defined at lines 69-84) employs regex-based pattern matching to identify ATX-style headings (e.g., `# Chapter 1`) and Setext-style headings (e.g., `Chapter II`).

This function returns a list of tuples containing the chapter number (or `None` if unparsable), the heading string, and the body text. By relying on pure-Python string processing rather than machine learning for segmentation, the chapter boundaries remain deterministic and reproducible across identical inputs.

### Stage 3: LLM Summary Generation and Validation

For each detected chapter tuple, the system constructs a metadata chunk that must contain two fields: a `description` (the chapter heading) and a `summary` (the LLM-generated abstract). An agent framework consumes the chapter body and assembles a prompt such as *"Summarise the following chapter in a concise paragraph"*, then queries the LLM (e.g., GPT-4) to produce the abstract.

The validation layer in [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py) (lines 36-38) enforces data integrity through a strict type guard: `if not isinstance(chunk, dict)...`. This validation confirms that every metadata chunk contains a string `summary` field before the pipeline finalizes. The completed metadata structure, including all per-chapter summaries, is written to `OUTPUT_META` (defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)) as [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json), which later integrates into the final [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) artifact.

## End-to-End Implementation Example

The following Python code demonstrates the complete workflow from extraction to serialized metadata, mirroring the actual implementation in the repository:

```python

# 1️⃣ Extract the book (CLI entry-point)

from book_to_skill.cli import main
main()                     # writes full_text.txt and metadata.json

# 2️⃣ Split the extracted text into chapters

from tools.discovery_tax import split_chapters
import book_to_skill.config

with open(book_to_skill.config.OUTPUT_TEXT, "r", encoding="utf-8") as f:
    full_text = f.read()

chapters = split_chapters(full_text)   # → [(1, "# Chapter 1", "..."), ...]

# 3️⃣ Generate a summary for each chapter (handled by the agent framework)

from book_to_skill.llm import generate_summary

metadata = []
for num, heading, body in chapters:
    summary = generate_summary(body)            # LLM completion

    metadata.append({"description": heading, "summary": summary})

# 4️⃣ Persist the metadata (validated later by paper_flat)

import json
import pathlib

path = pathlib.Path(book_to_skill.config.OUTPUT_META)
path.write_text(
    json.dumps({"chunks": metadata}, ensure_ascii=False), 
    encoding="utf-8"
)

```

## Summary

- **Book-to-Skill** generates per-chapter summaries through a deterministic three-stage pipeline implemented in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py), and [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py).
- **Chapter detection** relies on regex-based heuristics in the `split_chapters` function (lines 69-84 of [`discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/discovery_tax.py)) to identify headings and segment the full text into (number, heading, body) tuples.
- **Validation** occurs at [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py) (lines 36-38), where a type guard ensures every metadata chunk contains a valid string `summary` field before serialization.
- **Output** is written to [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json) (defined as `OUTPUT_META` in [`config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/config.py)), ensuring every chapter in the source document has a corresponding LLM-generated abstract in the final skill artifact.

## Frequently Asked Questions

### How does Book-to-Skill detect chapter boundaries in unstructured text?

Book-to-Skill uses the `split_chapters` function in [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) (lines 69-84) to apply regex-based heuristics that match ATX-style markdown headings (e.g., `# Chapter 1`) and Setext-style declarations (e.g., `Chapter II`). This deterministic approach returns a list of tuples containing the chapter number, heading text, and body content, ensuring consistent segmentation without machine-learning inference.

### Where are the generated per-chapter summaries stored?

The system writes per-chapter summaries to [`metadata.json`](https://github.com/virgiliojr94/book-to-skill/blob/main/metadata.json), located at the path defined by `OUTPUT_META` in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py). This file contains a JSON object with a `chunks` array, where each element includes a `description` (the chapter heading) and a `summary` (the LLM-generated abstract).

### What validation ensures every chapter includes a summary?

The validation logic in [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py) (lines 36-38) enforces schema compliance through a type guard that checks `if not isinstance(chunk, dict)`. This ensures every metadata chunk is a dictionary containing a string `summary` field, guaranteeing that the final [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) artifact includes a summary for every detected chapter.

### Which source files orchestrate the per-chapter summary workflow?

The workflow is coordinated across three primary files: [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) triggers the CLI and extraction logic; [`tools/discovery_tax.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/discovery_tax.py) handles chapter segmentation via `split_chapters`; and [`tools/evals/paper_flat.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/tools/evals/paper_flat.py) validates the output. Configuration constants like `OUTPUT_TEXT` and `OUTPUT_META` are defined in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py), establishing the temporary working directory structure.