Core Extraction Logic Functions in book-to-skill: A Complete Technical Guide

The core extraction logic functions in book-to-skill reside primarily in book_to_skill/utils.py, where parse_arguments, resolve_input_files, and extract_single_file orchestrate the conversion of documents into structured, token-aware text alongside metadata generation.

The open-source tool book-to-skill transforms arbitrary documents into AI-ready skill representations through a sophisticated extraction pipeline. Understanding the core extraction logic functions reveals how the tool handles everything from glob pattern resolution to magic-byte format detection and structural chapter analysis. This deep dive examines the central orchestration code in book_to_skill/utils.py and supporting modules that power the conversion process.

The Orchestration Layer in book_to_skill/utils.py

The heart of the extraction system lives in book_to_skill/utils.py (lines 17–171), which contains the primary functions for input handling and processing coordination.

CLI Argument Parsing with parse_arguments

Located at lines 17–50, parse_arguments(argv) normalizes command-line input into structured components. It processes the input paths, determines the extraction mode (technical or text), and handles the install-mode flag for dependency management via book_to_skill/dependencies.py.

File Discovery via resolve_input_files

The resolve_input_files(paths) function (lines 52–100) handles complex input scenarios. It expands glob patterns (such as ~/books/*.epub), walks directories recursively, normalizes tilde expansion for home directories, filters for supported extensions, and deduplicates results while preserving user-specified order.

The Single-File Extraction Pipeline

At lines 112–165, extract_single_file(input_path, extraction_mode, install_mode) serves as the central dispatch mechanism for document processing. This function validates files, performs format detection, manages external tool dependencies, and coordinates text extraction.

Format Detection and Validation

The function first validates file existence, then employs magic-byte sniffing to auto-detect the real format (PDF, EPUB, DOCX, etc.) regardless of file extension. It ensures required external tools are present through prepare_dependencies in book_to_skill/dependencies.py before invoking format-specific parsers.

Delegation to Format-Specific Parsers

Based on the detected format, extract_single_file delegates to specialized extractors located in book_to_skill/parsers/*.py. These include extract_html_file, extract_docx, extract_rtf, extract_with_pdftotext, and extract_with_ebooklib, each handling the unique structure of their respective document types.

Text Sanitization and Safety

After extraction, the raw text passes through sanitize_extracted_text in book_to_skill/sanitize.py (lines 139–151). This critical safety function strips invisible Unicode control characters and other unsafe code points before downstream processing, preventing encoding issues in the final output.

Token Estimation and Structure Analysis

The pipeline includes two analytical functions present in utils.py:

• estimate_tokens(text) (lines 84–100): Provides deterministic token-count heuristics handling both whitespace-delimited Latin scripts and CJK characters, essential for AI context window planning.

• detect_structure(text) (lines 72–106): Scans content for chapter numbering (numeric detection) or falls back to Markdown/AsciiDoc heading detection. It also searches for Table-of-Contents markers within the first 30,000 characters.

Main Execution and Aggregation

The main() entry point at lines 108–171 coordinates the end-to-end workflow. It prints the introductory banner, iterates over resolved input files calling extract_single_file, aggregates results into a combined text output (OUTPUT_TEXT) and JSON metadata (OUTPUT_META), and generates a final console report showing source counts, total pages, word counts, token estimates, and chapter detection results.

Practical Code Examples

Parsing Arguments and Resolving Files

# Example: emulate a CLI call

argv = ["book-to-skill", "~/books/*.pdf", "--mode", "technical"]
paths, mode, install = parse_arguments(argv)
files = resolve_input_files(paths)
print(f"Found {len(files)} PDFs → extraction mode: {mode}")

Extracting a Single File

from pathlib import Path

pdf_path = Path("/home/user/books/sample.pdf")
result = extract_single_file(pdf_path, extraction_mode="text", install_mode="ask")
print("Extraction method:", result["extraction_method"])
print("Detected chapters:", result["chapters_detected"])

Running the Full Pipeline

def run_demo():
    # Mimic a CLI: extract all supported files in a directory

    raw_paths = ["./sample_docs"]
    mode, install = "text", "ask"
    input_files = resolve_input_files(raw_paths)
    all_meta = []
    for p in input_files:
        meta = extract_single_file(p, mode, install)
        all_meta.append(meta)
    # Combine texts for downstream processing

    combined = "\n\n".join(meta["text"] for meta in all_meta)
    print("Combined length:", len(combined))
    # Show aggregated structure

    print(detect_structure(combined))

Summary

• The core extraction logic functions center on book_to_skill/utils.py, which contains the primary orchestration functions for the entire pipeline.
• parse_arguments and resolve_input_files handle CLI input normalization and file system discovery with glob expansion.
• extract_single_file manages the complete per-document workflow: validation, format detection, dependency installation, parsing, sanitization, and metadata generation.
• sanitize_extracted_text in sanitize.py ensures Unicode safety by stripping invisible control characters before processing.
• estimate_tokens and detect_structure provide AI-relevant analytics for context planning and document navigation.
• The main() function aggregates individual results into unified text and JSON metadata outputs suitable for downstream AI training.

Frequently Asked Questions

What is the entry point for the book-to-skill extraction pipeline?

The entry point is the main() function in book_to_skill/utils.py (lines 108–171). This function initializes the CLI banner, parses arguments via parse_arguments, resolves input files through resolve_input_files, and iterates over each document to aggregate results into the final output files OUTPUT_TEXT and OUTPUT_META.

How does book-to-skill detect document formats?

According to the source code in book_to_skill/utils.py, the tool uses magic-byte sniffing within extract_single_file (lines 112–165) to identify the real format of files regardless of their extensions. This detection happens before selecting the appropriate parser from the book_to_skill/parsers/ directory, ensuring PDFs, EPUBs, DOCX files, and other formats are processed by the correct extraction logic.

Where does the text sanitization occur in the extraction process?

Text sanitization occurs in sanitize_extracted_text located in book_to_skill/sanitize.py (lines 139–151). This function is called within the extract_single_file pipeline after format-specific parsing but before metadata generation, stripping invisible Unicode control characters to ensure clean output safe for AI processing.

How does the tool estimate token counts for extracted content?

The estimate_tokens(text) function in book_to_skill/utils.py (lines 84–100) implements a deterministic heuristic that accounts for both whitespace-delimited words in Latin scripts and character-based counting for CJK (Chinese, Japanese, Korean) text. This provides accurate token estimates essential for planning AI model context limits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}