How the Deterministic Extractor in book-to-skill Works: A Complete Technical Breakdown

The deterministic extractor in book-to-skill is a modular Python pipeline that transforms raw source files into reproducible markdown chapters through six pure-function stages: input resolution, format detection, dependency probing, text extraction, Unicode sanitization, and deterministic chunking.

The book-to-skill repository splits its architecture into two distinct halves: a deterministic extractor that produces clean structured text locally, and a spec-driven generator that assembles the final SKILL.md package. This guide examines the extractor component, which operates entirely without external services to guarantee identical outputs for identical inputs and dependency versions.

Understanding the Six-Stage Extraction Pipeline

The extractor orchestrates six reproducible stages defined in scripts/extract.py and its supporting modules. Each stage is implemented as a pure function to eliminate non-determinism and ensure the same input always produces the same directory of markdown files.

Stage 1: Input Resolution via resolve_sources()

The pipeline begins in book_to_skill/utils.py where resolve_sources() accepts single files, directories, or glob patterns. This function expands patterns and returns a reproducibly-ordered list of source paths, ensuring extraction order remains consistent across macOS, Linux, and Windows filesystems.

Stage 2: Format Detection with detect_format()

Located in book_to_skill/parsers/__init__.py, the detect_format() function identifies file types from extensions including PDF, EPUB, DOCX, HTML, RTF, TXT, and MD. This mapping determines which specific parser module the pipeline will invoke for raw extraction.

Stage 3: Dependency Probing in dependencies.py

Before extraction begins, book_to_skill/dependencies.py probes for optional external tools using a priority-based fallback system. For PDF documents, the extractor attempts docling first (preserving tables and code blocks for technical PDFs), then pdftotext for speed, followed by pure-Python fallbacks pypdf and pdfminer.six. The module executes --version checks or import attempts, selecting the first available tool and printing install commands via --check if none are found.

Stage 4: Raw Text Extraction

Each parser module in book_to_skill/parsers/ exposes a uniform extract() function returning raw text with metadata (filename, page numbers, headings). The PDF parser in book_to_skill/parsers/pdf.py routes to the selected tool, while book_to_skill/parsers/epub.py triangulates between ebooklib plus beautifulsoup4 or the standard library zipfile. Similar tiered logic exists in docx.py, html.py, rtf.py, text.py, and calibre.py.

Stage 5: Unicode Sanitization via sanitize.py

Before writing any output, all text passes through book_to_skill/sanitize.py. This stage removes BIDI control characters and other invisible Unicode that could corrupt downstream agents, normalizes line endings to LF, balances markdown code fences (triple back-ticks), and strips HTML boiler-plate and PDF page-number headers/footers via regex. These steps guarantee deterministic output regardless of the source file's creation OS or generation tool.

Stage 6: Deterministic Chunking and Output Generation

The final stage in book_to_skill/utils.py uses chunk_into_chapters() to split sanitized text by detecting markdown headers, reStructuredText sections, AsciiDoc markers, or "Chapter X" regex patterns. It calls build_glossary() and builds a patterns index and cheatsheet, then writes the deterministic directory structure containing chapters/, glossary.md, patterns.md, and metadata files ready for the generator.

Parser Priority Strategies by Format

The extractor maintains distinct priority lists for each format in book_to_skill/dependencies.py, optimizing for accuracy versus speed based on document classification.

PDF Processing Hierarchy

When the CLI marks a book as technical (via interactive prompt), the extractor prefers docling to preserve tables and code blocks. For text-heavy documents, it selects pdftotext for processing speed. The pure-Python fallbacks pypdf and pdfminer.six ensure extraction succeeds even in restricted environments without external binaries.

EPUB and DOCX Handling

EPUB parsing in book_to_skill/parsers/epub.py prioritizes ebooklib with beautifulsoup4 for robust HTML traversal, falling back to zipfile for pure-Python extraction. DOCX files use python-docx initially, with a raw ZIP/XML fallback for environments lacking the package. HTML parsing similarly prefers beautifulsoup4 before resorting to the standard library parser.

Practical Usage Examples

You can invoke the deterministic extractor via the command-line wrapper or embed it directly in Python applications.

Command-Line Interface

The scripts/extract.py CLI provides the primary entrypoint with interactive prompts for technical versus text-heavy classification:

python3 scripts/extract.py path/to/technical-book.pdf

Python Library API

Developers can embed the extractor directly using functions from book_to_skill/utils.py:

from book_to_skill.utils import resolve_sources, extract_file
from pathlib import Path

sources = resolve_sources(['my-book.pdf', 'chapters/'])
for src in sources:
    doc = extract_file(src)  # Returns dict with 'chapters', 'glossary', 'patterns'

    for chapter in doc['chapters']:
        output_path = Path(f'chapters/{chapter["slug"]}.md')
        output_path.write_text(chapter['content'])

This programmatic approach allows integration into larger document processing workflows while maintaining the same deterministic guarantees as the CLI.

Summary

• The deterministic extractor operates as six pure-function stages from input resolution to chunking, implemented across scripts/extract.py and book_to_skill/utils.py.
• book_to_skill/dependencies.py implements priority-based tool selection with graceful fallbacks to pure-Python parsers when external tools are unavailable.
• Sanitization in book_to_skill/sanitize.py removes non-deterministic Unicode, BIDI controls, and OS-specific line endings that could vary between runs.
• All parsers in book_to_skill/parsers/ expose uniform extract() interfaces, with format-specific logic for PDF, EPUB, DOCX, HTML, RTF, and plain text.
• Output is fully reproducible: given identical source files and the same versions of optional tools, the extractor emits identical chapters/, glossary.md, and metadata on every run.

Frequently Asked Questions

What makes the extractor "deterministic"?

The extractor produces bit-for-bit identical output for identical inputs because it uses pure functions for all transformations, probes tools in fixed priority order, normalizes Unicode and line endings, and contains no random seeds or network calls. As implemented in book_to_skill/sanitize.py, the pipeline strips BIDI control characters and OS-specific encodings that could introduce variance between runs.

Which PDF extraction tool does book-to-skill prefer?

For technical documents requiring table and code block preservation, docling is preferred. For speed on text-heavy PDFs, pdftotext is selected. Pure-Python fallbacks pypdf and pdfminer.six ensure the pipeline completes even without external dependencies. The selection logic lives in book_to_skill/dependencies.py.

Can I use the extractor without installing external tools like pdftotext?

Yes. The dependency probe automatically falls back to pure-Python implementations such as pypdf for PDFs and zipfile for EPUBs if external tools are unavailable, though extraction fidelity for complex layouts may decrease compared to docling or pdftotext.

How does the extractor handle different document structures?

The chunk_into_chapters() function in book_to_skill/utils.py detects structural elements using markdown headers (ATX and Setext), reStructuredText underlines, AsciiDoc markers, or regex patterns matching "Chapter X" strings. This creates a logical table of contents and glossary regardless of the original document's formatting style.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}