How Book-to-Skill Generates Per-Chapter Summaries: A Technical Deep Dive

Book-to-Skill generates per-chapter summaries through a three-stage pipeline that extracts raw text from PDFs or EPUBs, splits content using deterministic heuristics in tools/discovery_tax.py, and validates LLM-generated abstracts via tools/evals/paper_flat.py to ensure every chapter has a structured summary in metadata.json.

The virgiliojr94/book-to-skill open-source project transforms static books into actionable skill artifacts by creating structured per-chapter summaries for each section of a document. This process combines deterministic text processing with LLM-powered abstraction to convert unstructured book formats—PDF, EPUB, DOCX, and HTML—into validated JSON metadata. Understanding how Book-to-Skill generates these per-chapter summaries reveals a tightly-coupled pipeline of extraction, detection, and validation.

The Three-Stage Pipeline for Per-Chapter Summaries

Book-to-Skill processes books through three tightly-coupled stages that convert raw binary documents into structured, chapter-level metadata.

Stage 1: Text Extraction

The pipeline begins when scripts/extract.py invokes the CLI entry point in book_to_skill/cli.py. This component calls the appropriate parser for the input format—whether PDF, EPUB, HTML, or DOCX—and writes the extracted plain text to a temporary working directory.

According to the source code in book_to_skill/config.py, the system defines OUTPUT_TEXT as the destination for the full book content, typically located at book_skill_work-<pid>/full_text.txt. This file contains the complete, unprocessed text of the source document, serving as the input for the subsequent chapter detection phase.

Stage 2: Chapter Detection with Heuristics

Once extraction completes, the system parses the monolithic text file into discrete chapters using deterministic heuristics implemented in tools/discovery_tax.py. The core function split_chapters(text: str) → list[tuple[int | None, str, str]] (defined at lines 69-84) employs regex-based pattern matching to identify ATX-style headings (e.g., # Chapter 1) and Setext-style headings (e.g., Chapter II).

This function returns a list of tuples containing the chapter number (or None if unparsable), the heading string, and the body text. By relying on pure-Python string processing rather than machine learning for segmentation, the chapter boundaries remain deterministic and reproducible across identical inputs.

Stage 3: LLM Summary Generation and Validation

For each detected chapter tuple, the system constructs a metadata chunk that must contain two fields: a description (the chapter heading) and a summary (the LLM-generated abstract). An agent framework consumes the chapter body and assembles a prompt such as "Summarise the following chapter in a concise paragraph", then queries the LLM (e.g., GPT-4) to produce the abstract.

The validation layer in tools/evals/paper_flat.py (lines 36-38) enforces data integrity through a strict type guard: if not isinstance(chunk, dict).... This validation confirms that every metadata chunk contains a string summary field before the pipeline finalizes. The completed metadata structure, including all per-chapter summaries, is written to OUTPUT_META (defined in book_to_skill/config.py) as metadata.json, which later integrates into the final SKILL.md artifact.

End-to-End Implementation Example

The following Python code demonstrates the complete workflow from extraction to serialized metadata, mirroring the actual implementation in the repository:

# 1️⃣ Extract the book (CLI entry-point)

from book_to_skill.cli import main
main()                     # writes full_text.txt and metadata.json

# 2️⃣ Split the extracted text into chapters

from tools.discovery_tax import split_chapters
import book_to_skill.config

with open(book_to_skill.config.OUTPUT_TEXT, "r", encoding="utf-8") as f:
    full_text = f.read()

chapters = split_chapters(full_text)   # → [(1, "# Chapter 1", "..."), ...]

# 3️⃣ Generate a summary for each chapter (handled by the agent framework)

from book_to_skill.llm import generate_summary

metadata = []
for num, heading, body in chapters:
    summary = generate_summary(body)            # LLM completion

    metadata.append({"description": heading, "summary": summary})

# 4️⃣ Persist the metadata (validated later by paper_flat)

import json
import pathlib

path = pathlib.Path(book_to_skill.config.OUTPUT_META)
path.write_text(
    json.dumps({"chunks": metadata}, ensure_ascii=False), 
    encoding="utf-8"
)

Summary

• Book-to-Skill generates per-chapter summaries through a deterministic three-stage pipeline implemented in scripts/extract.py, tools/discovery_tax.py, and tools/evals/paper_flat.py.
• Chapter detection relies on regex-based heuristics in the split_chapters function (lines 69-84 of discovery_tax.py) to identify headings and segment the full text into (number, heading, body) tuples.
• Validation occurs at tools/evals/paper_flat.py (lines 36-38), where a type guard ensures every metadata chunk contains a valid string summary field before serialization.
• Output is written to metadata.json (defined as OUTPUT_META in config.py), ensuring every chapter in the source document has a corresponding LLM-generated abstract in the final skill artifact.

Frequently Asked Questions

How does Book-to-Skill detect chapter boundaries in unstructured text?

Book-to-Skill uses the split_chapters function in tools/discovery_tax.py (lines 69-84) to apply regex-based heuristics that match ATX-style markdown headings (e.g., # Chapter 1) and Setext-style declarations (e.g., Chapter II). This deterministic approach returns a list of tuples containing the chapter number, heading text, and body content, ensuring consistent segmentation without machine-learning inference.

Where are the generated per-chapter summaries stored?

The system writes per-chapter summaries to metadata.json, located at the path defined by OUTPUT_META in book_to_skill/config.py. This file contains a JSON object with a chunks array, where each element includes a description (the chapter heading) and a summary (the LLM-generated abstract).

What validation ensures every chapter includes a summary?

The validation logic in tools/evals/paper_flat.py (lines 36-38) enforces schema compliance through a type guard that checks if not isinstance(chunk, dict). This ensures every metadata chunk is a dictionary containing a string summary field, guaranteeing that the final SKILL.md artifact includes a summary for every detected chapter.

Which source files orchestrate the per-chapter summary workflow?

The workflow is coordinated across three primary files: scripts/extract.py triggers the CLI and extraction logic; tools/discovery_tax.py handles chapter segmentation via split_chapters; and tools/evals/paper_flat.py validates the output. Configuration constants like OUTPUT_TEXT and OUTPUT_META are defined in book_to_skill/config.py, establishing the temporary working directory structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}