How the Python Extractor Works in Book‑to‑Skill: A Deep Dive into the Document Processing Pipeline

The Python extractor in Book‑to‑Skill is a modular document‑to‑text pipeline that resolves input files, detects formats, dispatches to specialized parsers, sanitizes output, and generates both consolidated plain‑text and machine‑readable metadata.

The Book‑to‑Skill open‑source project transforms books, technical manuals, and documents into structured skill data. At its heart lies the Python extractor — a carefully architected dispatcher that handles everything from path resolution to multi‑stage PDF parsing. This article explores exactly how that extractor operates, walking through the source code in virgiliojr94/book-to-skill.

Entry Points: From Shell Command to Core Logic

The extractor exposes two entry points: a development wrapper and the installed console script.

The Wrapper Script (scripts/extract.py)

For development and direct repository execution, scripts/extract.py performs minimal setup:

# scripts/extract.py — adds project root to PYTHONPATH, then calls CLI

import sys
from pathlib import Path
project_root = Path(__file__).parent.parent
sys.path.insert(0, str(project_root))

from book_to_skill.cli import main
if __name__ == "__main__":
    main()

This pattern ensures the package imports resolve correctly without requiring installation.

The Console Entry Point (book_to_skill/cli.py)

The pip‑installable entry point lives in book_to_skill/cli.py. Its sole responsibility is environment normalization:

# book_to_skill/cli.py

import sys
from book_to_skill.utils import main as utils_main

def main():
    sys.stdout.reconfigure(encoding="utf-8")
    sys.stderr.reconfigure(encoding="utf-8")
    utils_main()

The actual argument parsing and orchestration happen in book_to_skill/utils.py.

Argument Parsing and Input Resolution

The parse_arguments() function in book_to_skill/utils.py (lines 75‑106) defines the command‑line interface:

# Simulated argument structure from utils.py

args = parse_arguments(["book-to-skill", "~/books/*.pdf", "manual.epub", "--mode", "technical"])

# Returns: (list[Path], mode: str, install: str)

Key parameters:

• Input paths — any mix of files, directories, or glob patterns
• --mode — either technical (layout‑aware extraction) or text (plain extraction)
• --install-missing — yes, no, or ask for dependency handling

File Resolution (resolve_input_files)

Before extraction begins, resolve_input_files() (lines 122‑166) transforms user input into a validated, deduplicated list of absolute Path objects:

• Expands shell globs (*.epub) and tilde shortcuts (~/documents)
• Recursively scans directories for supported extensions
• Preserves user‑specified order while removing duplicates
• Validates against the extension whitelist in book_to_skill/config.py

The Core Extraction Dispatcher

extract_single_file() serves as the heart of the Python extractor. For each resolved file, it executes a six‑stage pipeline:

1. Format Detection

Format detection uses file suffix first, falling back to magic‑byte sniffing for ambiguous cases (PDF, EPUB, DOCX) when suffixes are missing or misleading.

2. Dependency Resolution

prepare_dependencies() checks for and optionally installs external tools required by the chosen parser — poppler-utils for PDF conversion, Calibre's ebook-convert for proprietary formats, or Python libraries via pip.

3. Parser Dispatch by Extension

The extractor routes to format‑specific parsers in book_to_skill/parsers/:

Extension	Parser Module	Method
.epub	parsers/epub.py	extract_with_ebooklib() → fallback extract_with_zipfile()
.pdf	parsers/pdf.py	Docling → pdftotext → pypdf → pdfminer.six
.txt, .md	parsers/text.py	read_text_file()
.html, .htm	parsers/html.py	extract_html_file()
.docx	parsers/docx.py	extract_docx()
.rtf	parsers/rtf.py	extract_rtf()
.mobi, .azw3, etc.	parsers/calibre.py	extract_with_ebook_convert()

This cascading approach for PDFs ensures maximum compatibility: layout‑aware extraction succeeds when dependencies exist, but the extractor degrades gracefully through increasingly available fallbacks.

4. Text Sanitization

Extracted text passes through sanitize_extracted_text() from book_to_skill/sanitize.py, which strips invisible Unicode control characters that could corrupt downstream processing or inflate token counts.

5. Token Estimation

estimate_tokens() (lines 80‑96) applies a deterministic heuristic distinguishing Latin‑word tokens from CJK character tokens, providing accurate cost estimates for LLM‑based skill generation without requiring external tokenizers.

6. Structure Detection

detect_structure() (lines 30‑73) analyzes the first ~30 KB for:

• Numeric or structural chapter headings
• Table‑of‑Contents patterns

This metadata helps downstream systems understand document organization.

Aggregation and Output Generation

The main() function in book_to_skill/utils.py (lines 98‑131) orchestrates the complete pipeline:

# Conceptual flow from utils.py

results = [extract_single_file(p, mode, install) for p in resolved_paths]
combined_text = "\n\n".join(
    f"{'='*40}\nSOURCE: {r['source_path']}\n{'='*40}\n\n{r['text']}"
    for r in results
)

# Writes: OUTPUT_TEXT (consolidated .txt), OUTPUT_META (JSON manifest)

Output files:

• OUTPUT_TEXT — human‑readable concatenation with clear source banners
• OUTPUT_META — machine‑friendly JSON with per‑file metadata and consolidated structure analysis

Security: Safe Output Directory Creation

Before writing any files, prepare_output_dir() (lines 94‑116) creates a private work directory with strict validation:

• Rejects symbolic links (prevents path traversal)
• Verifies directory ownership (prevents privilege escalation via world‑writable paths)

This defensive programming protects against tampering attacks in multi‑user environments.

Practical Usage Examples

Command‑Line Extraction

# Single PDF with layout‑aware extraction

book-to-skill mybook.pdf --mode technical

# Batch processing with automatic dependency installation

book-to-skill ~/library/*.epub ~/library/*.pdf --install-missing ask

# Mixed formats in a directory

book-to-skill ~/tech-books/ --mode text

Programmatic Invocation

from pathlib import Path
import sys
from book_to_skill.utils import main as utils_main, extract_single_file

# Simulate CLI execution

sys.argv = ["book-to-skill", "example.epub", "--mode", "text"]
utils_main()

# Direct single‑file extraction

from book_to_skill.utils import parse_arguments
paths, mode, install = parse_arguments(["book-to-skill", "manual.pdf"])
result = extract_single_file(paths[0], mode, install)

print(f"Characters: {len(result['text'])}")
print(f"Estimated tokens: {result['token_count']}")
print(f"Chapters detected: {result['chapters_detected']}")

Key Source Files in the Extractor Architecture

File	Responsibility
scripts/extract.py	Development entry point with PYTHONPATH setup
book_to_skill/cli.py	UTF‑8 console configuration, forwards to utils
book_to_skill/utils.py	Argument parsing, file resolution, extraction dispatcher, aggregation
book_to_skill/parsers/pdf.py	Multi‑stage PDF extraction (Docling → pdftotext → pypdf → pdfminer)
book_to_skill/parsers/epub.py	EPUB handling via ebooklib with zip fallback
book_to_skill/parsers/html.py	HTML text extraction
book_to_skill/parsers/docx.py	DOCX parsing
book_to_skill/parsers/rtf.py	RTF extraction
book_to_skill/parsers/calibre.py	Calibre ebook-convert integration
book_to_skill/sanitize.py	Unicode control character removal
book_to_skill/config.py	Extensions, output paths, token heuristics

Summary

The Book‑to‑Skill Python extractor implements a robust, modular document processing pipeline:

• Flexible input handling — globs, directories, and mixed formats resolve to validated path lists
• Intelligent format detection — suffix‑based routing with magic‑byte fallback
• Degraded extraction — cascading PDF parsers ensure maximum document coverage
• Safe execution — dependency management and directory permissions prevent attacks
• Rich metadata — token estimates and structural detection support downstream skill generation
• Dual output — human‑ readable concatenated text plus machine‑parseable JSON

Frequently Asked Questions

How does the extractor handle PDFs with complex layouts?

The Python extractor tries four parsers in sequence: first Docling for layout‑aware extraction, then pdftotext (Poppler), pypdf, and finally pdfminer.six. This cascade ensures extraction succeeds even when optional dependencies are missing, with quality degrading gracefully rather than failing.

Can I extract proprietary formats like Kindle .azw3 files?

Yes. The extractor delegates to Calibre's ebook-convert via book_to_skill/parsers/calibre.py. Set --install-missing ask to prompt for Calibre installation, or install it manually before running.

What security measures protect the output directory?

prepare_output_dir() in utils.py validates that the output path is not a symbolic link and that the directory is owned by the current user. These checks prevent symlink attacks and privilege escalation via world‑writable directories.

How accurate is the token count estimation?

The heuristic in estimate_tokens() distinguishes between space‑delimited Latin scripts and CJK characters, providing deterministic counts without external dependencies. While approximate, it reliably identifies documents exceeding typical LLM context windows.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}