# How the Python Extractor Works in Book‑to‑Skill: A Deep Dive into the Document Processing Pipeline

> Explore the Python extractor in Book-to-Skill. This deep dive details its modular document processing pipeline, from format detection to text and metadata generation.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-31

---

**The Python extractor in Book‑to‑Skill is a modular document‑to‑text pipeline that resolves input files, detects formats, dispatches to specialized parsers, sanitizes output, and generates both consolidated plain‑text and machine‑readable metadata.**

The Book‑to‑Skill open‑source project transforms books, technical manuals, and documents into structured skill data. At its heart lies the **Python extractor** — a carefully architected dispatcher that handles everything from path resolution to multi‑stage PDF parsing. This article explores exactly how that extractor operates, walking through the source code in `virgiliojr94/book-to-skill`.

## Entry Points: From Shell Command to Core Logic

The extractor exposes two entry points: a development wrapper and the installed console script.

### The Wrapper Script ([`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py))

For development and direct repository execution, [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) performs minimal setup:

```python

# scripts/extract.py — adds project root to PYTHONPATH, then calls CLI

import sys
from pathlib import Path
project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))

from book_to_skill.cli import main
if __name__ == "__main__":
    main()

```

This pattern ensures the package imports resolve correctly without requiring installation.

### The Console Entry Point ([`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py))

The pip‑installable entry point lives in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py). Its sole responsibility is environment normalization:

```python

# book_to_skill/cli.py

import sys
from book_to_skill.utils import main as utils_main

def main():
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
    utils_main()

```

The actual argument parsing and orchestration happen in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py).

## Argument Parsing and Input Resolution

The `parse_arguments()` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 75‑106) defines the command‑line interface:

```python

# Simulated argument structure from utils.py

args = parse_arguments(["book-to-skill", "~/books/*.pdf", "manual.epub", "--mode", "technical"])

# Returns: (list[Path], mode: str, install: str)

```

**Key parameters:**
- **Input paths** — any mix of files, directories, or glob patterns
- **`--mode`** — either `technical` (layout‑aware extraction) or `text` (plain extraction)
- **`--install-missing`** — `yes`, `no`, or `ask` for dependency handling

### File Resolution (`resolve_input_files`)

Before extraction begins, `resolve_input_files()` (lines 122‑166) transforms user input into a validated, deduplicated list of absolute `Path` objects:

- Expands shell globs (`*.epub`) and tilde shortcuts (`~/documents`)
- Recursively scans directories for supported extensions
- Preserves user‑specified order while removing duplicates
- Validates against the extension whitelist in [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py)

## The Core Extraction Dispatcher

`extract_single_file()` serves as the heart of the Python extractor. For each resolved file, it executes a six‑stage pipeline:

### 1. Format Detection

Format detection uses file suffix first, falling back to **magic‑byte sniffing** for ambiguous cases (PDF, EPUB, DOCX) when suffixes are missing or misleading.

### 2. Dependency Resolution

`prepare_dependencies()` checks for and optionally installs external tools required by the chosen parser — `poppler-utils` for PDF conversion, Calibre's `ebook-convert` for proprietary formats, or Python libraries via pip.

### 3. Parser Dispatch by Extension

The extractor routes to format‑specific parsers in `book_to_skill/parsers/`:

| Extension | Parser Module | Method |
|-----------|---------------|--------|
| `.epub` | [`parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/epub.py) | `extract_with_ebooklib()` → fallback `extract_with_zipfile()` |
| `.pdf` | [`parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/pdf.py) | Docling → `pdftotext` → `pypdf` → `pdfminer.six` |
| `.txt`, `.md` | [`parsers/text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/text.py) | `read_text_file()` |
| `.html`, `.htm` | [`parsers/html.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/html.py) | `extract_html_file()` |
| `.docx` | [`parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/docx.py) | `extract_docx()` |
| `.rtf` | [`parsers/rtf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/rtf.py) | `extract_rtf()` |
| `.mobi`, `.azw3`, etc. | [`parsers/calibre.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/parsers/calibre.py) | `extract_with_ebook_convert()` |

This cascading approach for PDFs ensures **maximum compatibility**: layout‑aware extraction succeeds when dependencies exist, but the extractor degrades gracefully through increasingly available fallbacks.

### 4. Text Sanitization

Extracted text passes through `sanitize_extracted_text()` from [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py), which strips invisible Unicode control characters that could corrupt downstream processing or inflate token counts.

### 5. Token Estimation

`estimate_tokens()` (lines 80‑96) applies a deterministic heuristic distinguishing **Latin‑word tokens** from **CJK character tokens**, providing accurate cost estimates for LLM‑based skill generation without requiring external tokenizers.

### 6. Structure Detection

`detect_structure()` (lines 30‑73) analyzes the first ~30 KB for:
- Numeric or structural chapter headings
- Table‑of‑Contents patterns

This metadata helps downstream systems understand document organization.

## Aggregation and Output Generation

The `main()` function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 98‑131) orchestrates the complete pipeline:

```python

# Conceptual flow from utils.py

results = [extract_single_file(p, mode, install) for p in resolved_paths]
combined_text = "\n\n".join(
    f"{'='*40}\nSOURCE: {r['source_path']}\n{'='*40}\n\n{r['text']}"
    for r in results
)

# Writes: OUTPUT_TEXT (consolidated .txt), OUTPUT_META (JSON manifest)

```

**Output files:**
- **`OUTPUT_TEXT`** — human‑readable concatenation with clear source banners
- **`OUTPUT_META`** — machine‑friendly JSON with per‑file metadata and consolidated structure analysis

## Security: Safe Output Directory Creation

Before writing any files, `prepare_output_dir()` (lines 94‑116) creates a **private work directory** with strict validation:
- Rejects symbolic links (prevents path traversal)
- Verifies directory ownership (prevents privilege escalation via world‑writable paths)

This defensive programming protects against tampering attacks in multi‑user environments.

## Practical Usage Examples

### Command‑Line Extraction

```bash

# Single PDF with layout‑aware extraction

book-to-skill mybook.pdf --mode technical

# Batch processing with automatic dependency installation

book-to-skill ~/library/*.epub ~/library/*.pdf --install-missing ask

# Mixed formats in a directory

book-to-skill ~/tech-books/ --mode text

```

### Programmatic Invocation

```python
from pathlib import Path
import sys
from book_to_skill.utils import main as utils_main, extract_single_file

# Simulate CLI execution

sys.argv = ["book-to-skill", "example.epub", "--mode", "text"]
utils_main()

# Direct single‑file extraction

from book_to_skill.utils import parse_arguments
paths, mode, install = parse_arguments(["book-to-skill", "manual.pdf"])
result = extract_single_file(paths[0], mode, install)

print(f"Characters: {len(result['text'])}")
print(f"Estimated tokens: {result['token_count']}")
print(f"Chapters detected: {result['chapters_detected']}")

```

## Key Source Files in the Extractor Architecture

| File | Responsibility |
|------|--------------|
| [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) | Development entry point with PYTHONPATH setup |
| [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) | UTF‑8 console configuration, forwards to utils |
| [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Argument parsing, file resolution, extraction dispatcher, aggregation |
| [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py) | Multi‑stage PDF extraction (Docling → pdftotext → pypdf → pdfminer) |
| [`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py) | EPUB handling via ebooklib with zip fallback |
| [`book_to_skill/parsers/html.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/html.py) | HTML text extraction |
| [`book_to_skill/parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/docx.py) | DOCX parsing |
| [`book_to_skill/parsers/rtf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/rtf.py) | RTF extraction |
| [`book_to_skill/parsers/calibre.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/calibre.py) | Calibre `ebook-convert` integration |
| [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) | Unicode control character removal |
| [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) | Extensions, output paths, token heuristics |

## Summary

The Book‑to‑Skill Python extractor implements a **robust, modular document processing pipeline**:

- **Flexible input handling** — globs, directories, and mixed formats resolve to validated path lists
- **Intelligent format detection** — suffix‑based routing with magic‑byte fallback
- **Degraded extraction** — cascading PDF parsers ensure maximum document coverage
- **Safe execution** — dependency management and directory permissions prevent attacks
- **Rich metadata** — token estimates and structural detection support downstream skill generation
- **Dual output** — human‑ readable concatenated text plus machine‑parseable JSON

## Frequently Asked Questions

### How does the extractor handle PDFs with complex layouts?

The Python extractor tries **four parsers in sequence**: first Docling for layout‑aware extraction, then `pdftotext` (Poppler), `pypdf`, and finally `pdfminer.six`. This cascade ensures extraction succeeds even when optional dependencies are missing, with quality degrading gracefully rather than failing.

### Can I extract proprietary formats like Kindle `.azw3` files?

Yes. The extractor delegates to **Calibre's `ebook-convert`** via [`book_to_skill/parsers/calibre.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/calibre.py). Set `--install-missing ask` to prompt for Calibre installation, or install it manually before running.

### What security measures protect the output directory?

`prepare_output_dir()` in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py) validates that the output path is not a symbolic link and that the directory is owned by the current user. These checks prevent symlink attacks and privilege escalation via world‑writable directories.

### How accurate is the token count estimation?

The heuristic in `estimate_tokens()` distinguishes between space‑delimited Latin scripts and CJK characters, providing deterministic counts without external dependencies. While approximate, it reliably identifies documents exceeding typical LLM context windows.