# Core Extraction Logic Functions in book-to-skill: A Complete Technical Guide

> Explore core extraction logic functions in book-to-skill, detailed in utils.py. Discover how parse_arguments, resolve_input_files, and extract_single_file transform documents into token-aware text with metadata.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-26

---

**The core extraction logic functions in book-to-skill reside primarily in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), where `parse_arguments`, `resolve_input_files`, and `extract_single_file` orchestrate the conversion of documents into structured, token-aware text alongside metadata generation.**

The open-source tool **book-to-skill** transforms arbitrary documents into AI-ready skill representations through a sophisticated extraction pipeline. Understanding the core extraction logic functions reveals how the tool handles everything from glob pattern resolution to magic-byte format detection and structural chapter analysis. This deep dive examines the central orchestration code in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) and supporting modules that power the conversion process.

## The Orchestration Layer in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py)

The heart of the extraction system lives in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 17–171), which contains the primary functions for input handling and processing coordination.

### CLI Argument Parsing with `parse_arguments`

Located at lines 17–50, `parse_arguments(argv)` normalizes command-line input into structured components. It processes the input paths, determines the extraction mode (`technical` or `text`), and handles the install-mode flag for dependency management via [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py).

### File Discovery via `resolve_input_files`

The `resolve_input_files(paths)` function (lines 52–100) handles complex input scenarios. It expands glob patterns (such as `~/books/*.epub`), walks directories recursively, normalizes tilde expansion for home directories, filters for supported extensions, and deduplicates results while preserving user-specified order.

## The Single-File Extraction Pipeline

At lines 112–165, `extract_single_file(input_path, extraction_mode, install_mode)` serves as the central dispatch mechanism for document processing. This function validates files, performs format detection, manages external tool dependencies, and coordinates text extraction.

### Format Detection and Validation

The function first validates file existence, then employs **magic-byte sniffing** to auto-detect the real format (PDF, EPUB, DOCX, etc.) regardless of file extension. It ensures required external tools are present through `prepare_dependencies` in [`book_to_skill/dependencies.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/dependencies.py) before invoking format-specific parsers.

### Delegation to Format-Specific Parsers

Based on the detected format, `extract_single_file` delegates to specialized extractors located in `book_to_skill/parsers/*.py`. These include `extract_html_file`, `extract_docx`, `extract_rtf`, `extract_with_pdftotext`, and `extract_with_ebooklib`, each handling the unique structure of their respective document types.

### Text Sanitization and Safety

After extraction, the raw text passes through `sanitize_extracted_text` in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) (lines 139–151). This critical safety function strips invisible Unicode control characters and other unsafe code points before downstream processing, preventing encoding issues in the final output.

### Token Estimation and Structure Analysis

The pipeline includes two analytical functions present in [`utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/utils.py):

- **`estimate_tokens(text)`** (lines 84–100): Provides deterministic token-count heuristics handling both whitespace-delimited Latin scripts and CJK characters, essential for AI context window planning.

- **`detect_structure(text)`** (lines 72–106): Scans content for chapter numbering (numeric detection) or falls back to Markdown/AsciiDoc heading detection. It also searches for Table-of-Contents markers within the first 30,000 characters.

## Main Execution and Aggregation

The `main()` entry point at lines 108–171 coordinates the end-to-end workflow. It prints the introductory banner, iterates over resolved input files calling `extract_single_file`, aggregates results into a combined text output (`OUTPUT_TEXT`) and JSON metadata (`OUTPUT_META`), and generates a final console report showing source counts, total pages, word counts, token estimates, and chapter detection results.

## Practical Code Examples

### Parsing Arguments and Resolving Files

```python

# Example: emulate a CLI call

argv = ["book-to-skill", "~/books/*.pdf", "--mode", "technical"]
paths, mode, install = parse_arguments(argv)
files = resolve_input_files(paths)
print(f"Found {len(files)} PDFs → extraction mode: {mode}")

```

### Extracting a Single File

```python
from pathlib import Path

pdf_path = Path("/home/user/books/sample.pdf")
result = extract_single_file(pdf_path, extraction_mode="text", install_mode="ask")
print("Extraction method:", result["extraction_method"])
print("Detected chapters:", result["chapters_detected"])

```

### Running the Full Pipeline

```python
def run_demo():
    # Mimic a CLI: extract all supported files in a directory

    raw_paths = ["./sample_docs"]
    mode, install = "text", "ask"
    input_files = resolve_input_files(raw_paths)
    all_meta = []
    for p in input_files:
        meta = extract_single_file(p, mode, install)
        all_meta.append(meta)
    # Combine texts for downstream processing

    combined = "\n\n".join(meta["text"] for meta in all_meta)
    print("Combined length:", len(combined))
    # Show aggregated structure

    print(detect_structure(combined))

```

## Summary

- The **core extraction logic functions** center on [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), which contains the primary orchestration functions for the entire pipeline.
- **`parse_arguments`** and **`resolve_input_files`** handle CLI input normalization and file system discovery with glob expansion.
- **`extract_single_file`** manages the complete per-document workflow: validation, format detection, dependency installation, parsing, sanitization, and metadata generation.
- **`sanitize_extracted_text`** in [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) ensures Unicode safety by stripping invisible control characters before processing.
- **`estimate_tokens`** and **`detect_structure`** provide AI-relevant analytics for context planning and document navigation.
- The **`main()`** function aggregates individual results into unified text and JSON metadata outputs suitable for downstream AI training.

## Frequently Asked Questions

### What is the entry point for the book-to-skill extraction pipeline?

The entry point is the **`main()`** function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 108–171). This function initializes the CLI banner, parses arguments via `parse_arguments`, resolves input files through `resolve_input_files`, and iterates over each document to aggregate results into the final output files `OUTPUT_TEXT` and `OUTPUT_META`.

### How does book-to-skill detect document formats?

According to the source code in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py), the tool uses **magic-byte sniffing** within `extract_single_file` (lines 112–165) to identify the real format of files regardless of their extensions. This detection happens before selecting the appropriate parser from the `book_to_skill/parsers/` directory, ensuring PDFs, EPUBs, DOCX files, and other formats are processed by the correct extraction logic.

### Where does the text sanitization occur in the extraction process?

Text sanitization occurs in **`sanitize_extracted_text`** located in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) (lines 139–151). This function is called within the `extract_single_file` pipeline after format-specific parsing but before metadata generation, stripping invisible Unicode control characters to ensure clean output safe for AI processing.

### How does the tool estimate token counts for extracted content?

The **`estimate_tokens(text)`** function in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) (lines 84–100) implements a deterministic heuristic that accounts for both whitespace-delimited words in Latin scripts and character-based counting for CJK (Chinese, Japanese, Korean) text. This provides accurate token estimates essential for planning AI model context limits.