How the Deterministic Python Extractor Works in book-to-skill: A Deep Dive into the 9-Step Pipeline

The deterministic Python extractor in book-to-skill processes documents through a pure-Python, stateless pipeline that guarantees reproducible results by using ordered processing, no randomness, and no external caches.

The book-to-skill repository provides a reliable tool for converting books and documents into structured text suitable for skill generation. Its deterministic Python extractor forms the core processing engine, ensuring that identical inputs always produce identical outputs. This article examines the complete extraction pipeline as implemented in the virgiliojr94/book-to-skill source code.

CLI Bootstrap and Entry Point

Execution begins in scripts/extract.py, which prepares the Python environment before handing control to the main logic.

# scripts/extract.py (lines 19-26)

import sys
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from book_to_skill.cli import main
main()

This wrapper forces UTF-8 I/O and injects the project root into sys.path, ensuring consistent behavior across platforms.

Argument Parsing and Mode Selection

The parse_arguments() function in book_to_skill/utils.py handles CLI options including --mode, --install-missing, and input path collection. It normalizes the install mode through normalize_install_mode (lines 75-89).

Supported extraction modes include:

• "technical" — tries Docling first, falls back to pdftotext
• "text" — prioritizes plain text extraction tools

Input File Resolution with Deterministic Ordering

The resolve_input_files() function (lines 111-168 in utils.py) guarantees reproducible file ordering:

# From book_to_skill/utils.py

def resolve_input_files(paths):
    """Expands globs, walks directories, and returns deduplicated,
    deterministically sorted files."""
    files = []
    for p in paths:
        if p.is_dir():
            # Recursively collect with sorted, case-insensitive ordering

            files.extend(sorted(p.rglob("*"), key=lambda x: str(x).lower()))
        else:
            files.append(p)
    # Deduplicate while preserving user-specified order for explicit args

    seen = set()
    return [f for f in files if not (f in seen or seen.add(f))]

This ensures alphabetical sorting for glob and directory expansions, with explicit file arguments retaining their user-provided sequence.

Dependency Preparation and Validation

Before extraction, prepare_dependencies() (lines 221-229) verifies that external tools are available for the detected file type. If a required tool like ebook-convert or pdftotext is missing, it raises a clear ExtractionError rather than failing silently.

Per-File Extraction Orchestration

The heart of the system is extract_single_file() (lines 770-890), which:

1. Detects file type via extension or magic bytes
2. Dispatches to format-specific parsers (extract_with_ebooklib, extract_with_pdftotext, extract_docx, etc.)
3. Handles large EPUBs by dropping images when thresholds are exceeded
4. Records the extraction method used for transparency

# Example: Extract a single PDF file in technical mode

from pathlib import Path
from book_to_skill.utils import extract_single_file

result = extract_single_file(
    input_path=Path("example.pdf"),
    extraction_mode="technical",   # tries Docling first, falls back to pdftotext

    install_mode="ask"             # prompts to install missing dependencies

)

print(result["extraction_method"])   # → "docling" or "pdftotext"

print(result["estimated_tokens"])    # deterministic token estimate

Unicode Sanitization Against Hidden Attacks

Extracted text passes through sanitize_extracted_text() in book_to_skill/sanitize.py (lines 4-50), which strips:

• Zero-width characters (zero-width space, joiner, non-joiner)
• Bidirectional control characters (LTR/RTL overrides)
• Annotation marks and variation selectors

This hardens against prompt-injection attacks using invisible Unicode code points.

Deterministic Token Estimation

The estimate_tokens() function (lines 84-101) provides reproducible token counts without external dependencies:

def estimate_tokens(text: str) -> int:
    """Deterministic token estimation using script-aware heuristics."""
    if is_latin_script(text):
        # Word-based heuristic for Latin scripts

        return len(text.split())
    else:
        # Character-based for CJK and other scripts

        return len(text)

This pure-Python heuristic eliminates variance from model-based tokenizers.

Structure Detection for Skill Generation

detect_structure() (lines 221-273) analyzes the full text to identify:

• Chapter headings via multilingual regex patterns
• Markdown/AsciiDoc structural headings
• Table-of-Contents markers within the first 30,000 characters

This metadata informs downstream skill generation by segmenting content at logical boundaries.

Result Consolidation and Output

The CLI aggregates all extractions in book_to_skill/utils.py (lines 1100-1170), writing:

• OUTPUT_TEXT — merged extracted content
• Aggregated metadata — total size, page counts, token estimates, chapter counts, ToC presence

A short support note prints completion status.

Batch Processing Example

For processing multiple files deterministically:

# Example: Batch-process a directory with glob pattern

from book_to_skill.utils import resolve_input_files, parse_arguments

paths, mode, install = parse_arguments(
    ["--mode", "text", "books/*.epub"]
)
files = resolve_input_files(paths)

for f in files:
    data = extract_single_file(f, mode, install)
    # deterministic: same files always produce same results

    process_text(data["text"])

Key Source Files

File	Role
scripts/extract.py	CLI wrapper with UTF-8 enforcement
book_to_skill/utils.py	Core extraction pipeline and orchestration
book_to_skill/sanitize.py	Invisible Unicode character removal
book_to_skill/parsers/*	Format-specific extractors (PDF, EPUB, DOCX, HTML)
book_to_skill/config.py	Output locations and supported extensions
book_to_skill/dependencies.py	External tool availability checks
book_to_skill/cli.py	Entry-point wiring all components

Summary

• Determinism is achieved through pure-Python heuristics, ordered file processing, and stateless function design
• Security is hardened via Unicode sanitization that strips invisible control characters
• Flexibility supports multiple formats (PDF, EPUB, DOCX, HTML) with graceful fallback chains
• Transparency records which extraction method succeeded for each file
• Reproducibility guarantees identical results for identical inputs across all operations

Frequently Asked Questions

What makes the book-to-skill extractor "deterministic"?

The extractor contains no randomness: file ordering uses sorted, case-insensitive alphabetical ordering; token estimation uses script-aware heuristics rather than model-based tokenizers; and all extraction functions are stateless, operating only on their inputs. No external caches or mutable global state affect outputs.

How does the extractor handle missing dependencies for file formats?

The prepare_dependencies() function checks tool availability before extraction. With install_mode="ask", it prompts for permission to install missing tools; with "auto" it attempts automatic installation; with "skip" it raises ExtractionError for unsupported formats.

Why does the extractor sanitize invisible Unicode characters?

Invisible characters like zero-width spaces and bidirectional overrides can hide malicious instructions in extracted text. The sanitize_extracted_text() function in sanitize.py removes these to prevent prompt-injection attacks when the output feeds into LLM-based skill generation.

What token estimation method does book-to-skill use?

The estimate_tokens() function uses a simple, reproducible heuristic: word counts for Latin scripts and direct character counts for CJK scripts. This avoids non-determinism from neural tokenizers while providing sufficiently accurate estimates for most use cases.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}