# How the Deterministic Python Extractor Works in book-to-skill: A Deep Dive into the 9-Step Pipeline

> Discover how the deterministic Python extractor in book-to-skill ensures reproducible results through a 9-step stateless pipeline. Learn about its ordered processing and lack of randomness.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-09-01

---

**The deterministic Python extractor in book-to-skill processes documents through a pure-Python, stateless pipeline that guarantees reproducible results by using ordered processing, no randomness, and no external caches.**

The `book-to-skill` repository provides a reliable tool for converting books and documents into structured text suitable for skill generation. Its **deterministic Python extractor** forms the core processing engine, ensuring that identical inputs always produce identical outputs. This article examines the complete extraction pipeline as implemented in the virgiliojr94/book-to-skill source code.

## CLI Bootstrap and Entry Point

Execution begins in **[`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py)**, which prepares the Python environment before handing control to the main logic.

```python

# scripts/extract.py (lines 19-26)

import sys
sys.path.insert(0, str(Path(__file__).resolve().parent.parent))
from book_to_skill.cli import main
main()

```

This wrapper forces UTF-8 I/O and injects the project root into `sys.path`, ensuring consistent behavior across platforms.

## Argument Parsing and Mode Selection

The **`parse_arguments()`** function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) handles CLI options including `--mode`, `--install-missing`, and input path collection. It normalizes the install mode through **`normalize_install_mode`** (lines 75-89).

Supported extraction modes include:
- **`"technical"`** — tries Docling first, falls back to pdftotext
- **`"text"`** — prioritizes plain text extraction tools

## Input File Resolution with Deterministic Ordering

The **`resolve_input_files()`** function (lines 111-168 in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py)) guarantees reproducible file ordering:

```python

# From book_to_skill/utils.py

def resolve_input_files(paths):
    """Expands globs, walks directories, and returns deduplicated,
    deterministically sorted files."""
    files = []
    for p in paths:
        if p.is_dir():
            # Recursively collect with sorted, case-insensitive ordering

            files.extend(sorted(p.rglob("*"), key=lambda x: str(x).lower()))
        else:
            files.append(p)
    # Deduplicate while preserving user-specified order for explicit args

    seen = set()
    return [f for f in files if not (f in seen or seen.add(f))]

```

This ensures **alphabetical sorting** for glob and directory expansions, with explicit file arguments retaining their user-provided sequence.

## Dependency Preparation and Validation

Before extraction, **`prepare_dependencies()`** (lines 221-229) verifies that external tools are available for the detected file type. If a required tool like `ebook-convert` or `pdftotext` is missing, it raises a clear **`ExtractionError`** rather than failing silently.

## Per-File Extraction Orchestration

The heart of the system is **`extract_single_file()`** (lines 770-890), which:

1. **Detects file type** via extension or magic bytes
2. **Dispatches to format-specific parsers** (`extract_with_ebooklib`, `extract_with_pdftotext`, `extract_docx`, etc.)
3. **Handles large EPUBs** by dropping images when thresholds are exceeded
4. **Records the extraction method** used for transparency

```python

# Example: Extract a single PDF file in technical mode

from pathlib import Path
from book_to_skill.utils import extract_single_file

result = extract_single_file(
    input_path=Path("example.pdf"),
    extraction_mode="technical",   # tries Docling first, falls back to pdftotext

    install_mode="ask"             # prompts to install missing dependencies

)

print(result["extraction_method"])   # → "docling" or "pdftotext"

print(result["estimated_tokens"])    # deterministic token estimate

```

## Unicode Sanitization Against Hidden Attacks

Extracted text passes through **`sanitize_extracted_text()`** in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) (lines 4-50), which strips:

- **Zero-width characters** (zero-width space, joiner, non-joiner)
- **Bidirectional control characters** (LTR/RTL overrides)
- **Annotation marks** and **variation selectors**

This hardens against prompt-injection attacks using invisible Unicode code points.

## Deterministic Token Estimation

The **`estimate_tokens()`** function (lines 84-101) provides reproducible token counts without external dependencies:

```python
def estimate_tokens(text: str) -> int:
    """Deterministic token estimation using script-aware heuristics."""
    if is_latin_script(text):
        # Word-based heuristic for Latin scripts

        return len(text.split())
    else:
        # Character-based for CJK and other scripts

        return len(text)

```

This **pure-Python heuristic** eliminates variance from model-based tokenizers.

## Structure Detection for Skill Generation

**`detect_structure()`** (lines 221-273) analyzes the full text to identify:

- **Chapter headings** via multilingual regex patterns
- **Markdown/AsciiDoc structural headings**
- **Table-of-Contents markers** within the first 30,000 characters

This metadata informs downstream skill generation by segmenting content at logical boundaries.

## Result Consolidation and Output

The CLI aggregates all extractions in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 1100-1170), writing:
- **`OUTPUT_TEXT`** — merged extracted content
- **Aggregated metadata** — total size, page counts, token estimates, chapter counts, ToC presence

A short support note prints completion status.

## Batch Processing Example

For processing multiple files deterministically:

```python

# Example: Batch-process a directory with glob pattern

from book_to_skill.utils import resolve_input_files, parse_arguments

paths, mode, install = parse_arguments(
    ["--mode", "text", "books/*.epub"]
)
files = resolve_input_files(paths)

for f in files:
    data = extract_single_file(f, mode, install)
    # deterministic: same files always produce same results

    process_text(data["text"])

```

## Key Source Files

| File | Role |
|------|------|
| [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py) | CLI wrapper with UTF-8 enforcement |
| [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) | Core extraction pipeline and orchestration |
| [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) | Invisible Unicode character removal |
| `book_to_skill/parsers/*` | Format-specific extractors (PDF, EPUB, DOCX, HTML) |
| [`book_to_skill/config.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/config.py) | Output locations and supported extensions |
| [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) | External tool availability checks |
| [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) | Entry-point wiring all components |

## Summary

- **Determinism** is achieved through pure-Python heuristics, ordered file processing, and stateless function design
- **Security** is hardened via Unicode sanitization that strips invisible control characters
- **Flexibility** supports multiple formats (PDF, EPUB, DOCX, HTML) with graceful fallback chains
- **Transparency** records which extraction method succeeded for each file
- **Reproducibility** guarantees identical results for identical inputs across all operations

## Frequently Asked Questions

### What makes the book-to-skill extractor "deterministic"?

The extractor contains no randomness: file ordering uses sorted, case-insensitive alphabetical ordering; token estimation uses script-aware heuristics rather than model-based tokenizers; and all extraction functions are stateless, operating only on their inputs. No external caches or mutable global state affect outputs.

### How does the extractor handle missing dependencies for file formats?

The `prepare_dependencies()` function checks tool availability before extraction. With `install_mode="ask"`, it prompts for permission to install missing tools; with `"auto"` it attempts automatic installation; with `"skip"` it raises `ExtractionError` for unsupported formats.

### Why does the extractor sanitize invisible Unicode characters?

Invisible characters like zero-width spaces and bidirectional overrides can hide malicious instructions in extracted text. The `sanitize_extracted_text()` function in [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) removes these to prevent prompt-injection attacks when the output feeds into LLM-based skill generation.

### What token estimation method does book-to-skill use?

The `estimate_tokens()` function uses a simple, reproducible heuristic: word counts for Latin scripts and direct character counts for CJK scripts. This avoids non-determinism from neural tokenizers while providing sufficiently accurate estimates for most use cases.