# How the Deterministic Extractor in book-to-skill Works: A Complete Technical Breakdown

> Discover the deterministic extractor in book-to-skill. Learn how this Python pipeline converts source files into markdown chapters through six pure-function stages for reproducible results.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-26

---

**The deterministic extractor in book-to-skill is a modular Python pipeline that transforms raw source files into reproducible markdown chapters through six pure-function stages: input resolution, format detection, dependency probing, text extraction, Unicode sanitization, and deterministic chunking.**

The *book-to-skill* repository splits its architecture into two distinct halves: a deterministic extractor that produces clean structured text locally, and a spec-driven generator that assembles the final [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) package. This guide examines the extractor component, which operates entirely without external services to guarantee identical outputs for identical inputs and dependency versions.

## Understanding the Six-Stage Extraction Pipeline

The extractor orchestrates six reproducible stages defined in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) and its supporting modules. Each stage is implemented as a pure function to eliminate non-determinism and ensure the same input always produces the same directory of markdown files.

### Stage 1: Input Resolution via `resolve_sources()`

The pipeline begins in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) where `resolve_sources()` accepts single files, directories, or glob patterns. This function expands patterns and returns a reproducibly-ordered list of source paths, ensuring extraction order remains consistent across macOS, Linux, and Windows filesystems.

### Stage 2: Format Detection with `detect_format()`

Located in [`book_to_skill/parsers/__init__.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/__init__.py), the `detect_format()` function identifies file types from extensions including PDF, EPUB, DOCX, HTML, RTF, TXT, and MD. This mapping determines which specific parser module the pipeline will invoke for raw extraction.

### Stage 3: Dependency Probing in [`dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/dependencies.py)

Before extraction begins, [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) probes for optional external tools using a priority-based fallback system. For PDF documents, the extractor attempts **`docling`** first (preserving tables and code blocks for technical PDFs), then **`pdftotext`** for speed, followed by pure-Python fallbacks **`pypdf`** and **`pdfminer.six`**. The module executes `--version` checks or import attempts, selecting the first available tool and printing install commands via `--check` if none are found.

### Stage 4: Raw Text Extraction

Each parser module in `book_to_skill/parsers/` exposes a uniform `extract()` function returning raw text with metadata (filename, page numbers, headings). The PDF parser in [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py) routes to the selected tool, while [`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py) triangulates between `ebooklib` plus `beautifulsoup4` or the standard library `zipfile`. Similar tiered logic exists in [`docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/docx.py), [`html.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/html.py), [`rtf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/rtf.py), [`text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/text.py), and [`calibre.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/calibre.py).

### Stage 5: Unicode Sanitization via [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py)

Before writing any output, all text passes through [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py). This stage removes **BIDI control characters** and other invisible Unicode that could corrupt downstream agents, normalizes line endings to LF, balances markdown code fences (triple back-ticks), and strips HTML boiler-plate and PDF page-number headers/footers via regex. These steps guarantee deterministic output regardless of the source file's creation OS or generation tool.

### Stage 6: Deterministic Chunking and Output Generation

The final stage in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) uses `chunk_into_chapters()` to split sanitized text by detecting markdown headers, reStructuredText sections, AsciiDoc markers, or "Chapter X" regex patterns. It calls `build_glossary()` and builds a patterns index and cheatsheet, then writes the deterministic directory structure containing `chapters/`, [`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md), [`patterns.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/patterns.md), and metadata files ready for the generator.

## Parser Priority Strategies by Format

The extractor maintains distinct priority lists for each format in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py), optimizing for accuracy versus speed based on document classification.

### PDF Processing Hierarchy

When the CLI marks a book as *technical* (via interactive prompt), the extractor prefers `docling` to preserve tables and code blocks. For text-heavy documents, it selects `pdftotext` for processing speed. The pure-Python fallbacks `pypdf` and `pdfminer.six` ensure extraction succeeds even in restricted environments without external binaries.

### EPUB and DOCX Handling

EPUB parsing in [`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py) prioritizes `ebooklib` with `beautifulsoup4` for robust HTML traversal, falling back to `zipfile` for pure-Python extraction. DOCX files use `python-docx` initially, with a raw ZIP/XML fallback for environments lacking the package. HTML parsing similarly prefers `beautifulsoup4` before resorting to the standard library parser.

## Practical Usage Examples

You can invoke the deterministic extractor via the command-line wrapper or embed it directly in Python applications.

### Command-Line Interface

The [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) CLI provides the primary entrypoint with interactive prompts for technical versus text-heavy classification:

```bash
python3 scripts/extract.py path/to/technical-book.pdf

```

### Python Library API

Developers can embed the extractor directly using functions from [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py):

```python
from book_to_skill.utils import resolve_sources, extract_file
from pathlib import Path

sources = resolve_sources(['my-book.pdf', 'chapters/'])
for src in sources:
    doc = extract_file(src)  # Returns dict with 'chapters', 'glossary', 'patterns'

    for chapter in doc['chapters']:
        output_path = Path(f'chapters/{chapter["slug"]}.md')
        output_path.write_text(chapter['content'])

```

This programmatic approach allows integration into larger document processing workflows while maintaining the same deterministic guarantees as the CLI.

## Summary

- The deterministic extractor operates as six pure-function stages from input resolution to chunking, implemented across [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) and [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py).
- **[`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py)** implements priority-based tool selection with graceful fallbacks to pure-Python parsers when external tools are unavailable.
- Sanitization in **[`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py)** removes non-deterministic Unicode, BIDI controls, and OS-specific line endings that could vary between runs.
- All parsers in **`book_to_skill/parsers/`** expose uniform `extract()` interfaces, with format-specific logic for PDF, EPUB, DOCX, HTML, RTF, and plain text.
- Output is fully reproducible: given identical source files and the same versions of optional tools, the extractor emits identical `chapters/`, [`glossary.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/glossary.md), and metadata on every run.

## Frequently Asked Questions

### What makes the extractor "deterministic"?

The extractor produces bit-for-bit identical output for identical inputs because it uses pure functions for all transformations, probes tools in fixed priority order, normalizes Unicode and line endings, and contains no random seeds or network calls. As implemented in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), the pipeline strips BIDI control characters and OS-specific encodings that could introduce variance between runs.

### Which PDF extraction tool does book-to-skill prefer?

For technical documents requiring table and code block preservation, **`docling`** is preferred. For speed on text-heavy PDFs, **`pdftotext`** is selected. Pure-Python fallbacks **`pypdf`** and **`pdfminer.six`** ensure the pipeline completes even without external dependencies. The selection logic lives in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py).

### Can I use the extractor without installing external tools like pdftotext?

Yes. The dependency probe automatically falls back to pure-Python implementations such as `pypdf` for PDFs and `zipfile` for EPUBs if external tools are unavailable, though extraction fidelity for complex layouts may decrease compared to `docling` or `pdftotext`.

### How does the extractor handle different document structures?

The `chunk_into_chapters()` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) detects structural elements using markdown headers (ATX and Setext), reStructuredText underlines, AsciiDoc markers, or regex patterns matching "Chapter X" strings. This creates a logical table of contents and glossary regardless of the original document's formatting style.