How to Use the extract.py Script in book-to-skill

The extract.py script is a lightweight entry-point wrapper that adjusts the Python path and delegates execution to the main() function in book_to_skill/cli.py, initiating a full extraction pipeline that converts PDFs, EPUBs, DOCX files, and other documents into sanitized text and structured metadata.

The book-to-skill repository provides a document extraction toolkit designed to transform books and technical manuals into AI-ready formats. The extract.py script serves as the primary interface to this system, handling everything from dependency resolution to structural analysis while outputting standardized text files and JSON metadata.

What extract.py Actually Does

scripts/extract.py is intentionally minimal. When executed, it modifies sys.path to include the project root directory, ensuring the book_to_skill package can be imported correctly. It then calls the main() function defined in book_to_skill/cli.py【⁴⁰⁾⁴¹⁾】, transferring control to the core extraction logic.

This architectural separation keeps the entry point thin and maintainable while the heavy lifting occurs within the package modules.

Running the Extractor: Command-Line Examples

You can invoke the extraction pipeline using any of three equivalent methods:

./scripts/extract.py <path-or-glob> [options]
python3 -m book_to_skill <path-or-glob> [options]
python3 scripts/extract.py <path-or-glob> [options]

Basic Single-File Extraction

To extract a single document, provide the file path directly:

./scripts/extract.py mybook.pdf

Batch Processing with Glob Patterns

Process multiple files simultaneously using shell glob patterns:

./scripts/extract.py books/*.epub

Technical Mode for Complex Layouts

For PDFs with complex layouts, use the technical mode which leverages Docling for layout-aware extraction:

./scripts/extract.py report.pdf --mode technical

Handling Missing Dependencies

Automatically detect and install missing extractor dependencies with interactive confirmation:

./scripts/extract.py thesis.docx --install-missing ask

To view all available options:

./scripts/extract.py --help

Inside the Extraction Pipeline

The main() function in book_to_skill/cli.py orchestrates a five-stage pipeline that processes documents systematically.

Argument Parsing and Path Resolution

First, main() calls parse_arguments() from book_to_skill/utils.py【⁴³⁾】 to handle CLI flags like --mode and --install-missing. It then uses resolve_input_files()【⁴⁴⁾】 to expand file paths, directories, or glob patterns into a concrete list of target documents. The utility also calls prepare_output_dir()【⁴²⁾】 to ensure the OUTPUT_DIR exists before processing begins.

Document Processing and Text Extraction

For each resolved file, main() invokes extract_single_file(), which performs format detection and selects the appropriate parser from book_to_skill/parsers/ (e.g., pdf.py, epub.py). If --install-missing is specified, the system calls prepare_dependencies() from book_to_skill/dependencies.py【⁴⁵⁾】 to fetch required libraries before extraction proceeds.

The extraction engine handles multiple formats including PDF, EPUB, DOCX, TXT, HTML, and RTF, automatically selecting the best backend for each (such as pdfminer, pdftotext, or docling for PDFs).

Sanitization and Metadata Generation

After raw extraction, the pipeline passes text through sanitize_extracted_text() to clean encoding issues and formatting artifacts. It then runs estimate_tokens() to calculate token counts and detect_structure() to identify chapter boundaries and table-of-contents presence【⁴⁰⁾】.

All results are written to two files defined in book_to_skill/config.py:

• full_text.txt (OUTPUT_TEXT) containing the sanitized document text
• metadata.json (OUTPUT_META) containing token counts, page numbers, chapter counts, and structure detection results

The script concludes by printing a summary showing pages processed, word counts, estimated tokens, detected chapters, and TOC detection status.

Key Source Files in the Architecture

Understanding the codebase structure helps with customization and debugging:

• scripts/extract.py – The wrapper script that adjusts sys.path and launches main()
• book_to_skill/cli.py – Implements the CLI interface and the main extraction loop
• book_to_skill/utils.py – Contains parse_arguments(), resolve_input_files(), extract_single_file(), sanitize_extracted_text(), estimate_tokens(), and detect_structure()
• book_to_skill/parsers/ – Directory housing format-specific extractors for PDF, EPUB, DOCX, HTML, RTF, and other formats
• book_to_skill/dependencies.py – Manages optional dependency checking and installation via prepare_dependencies()
• book_to_skill/config.py – Defines constants including OUTPUT_DIR, OUTPUT_TEXT, and OUTPUT_META

Summary

• extract.py is a shim that bootstraps the environment and calls book_to_skill/cli.py:main() to start processing.
• Three invocation methods work identically: direct execution, module execution (-m), or explicit python3 calls.
• Multi-format support handles PDF, EPUB, DOCX, TXT, HTML, and RTF through specialized parsers in book_to_skill/parsers/.
• Dual output generates full_text.txt and metadata.json in the configured OUTPUT_DIR.
• Layout-aware extraction is available via --mode technical using the Docling backend.
• Dependency management can be automated with --install-missing ask or yes.

Frequently Asked Questions

Can I run book-to-skill without using extract.py?

Yes. Because extract.py is just a path-adjusting wrapper, you can invoke the package directly as a module using python3 -m book_to_skill <path> [options]. This executes the same main() function without requiring the shell script wrapper.

What file formats does extract.py support?

According to the source analysis, the system supports PDF, EPUB, DOCX, TXT, HTML, and RTF files. Each format has a dedicated parser in the book_to_skill/parsers/ directory, with PDFs supporting multiple extraction backends including pdfminer, pdftotext, and docling.

Where does extract.py save the extracted content?

Output files are written to the directory specified by OUTPUT_DIR in book_to_skill/config.py【⁴⁶⁾】. By default, the system generates full_text.txt (the sanitized document content) and metadata.json (containing token estimates, page counts, chapter detection, and TOC status). The script also prints a console summary of the extraction results.

How does the technical mode differ from standard extraction?

Standard extraction extracts raw text sequentially, while --mode technical activates layout-aware processing using Docling. This mode preserves structural information like columns, tables, and formatting hierarchies, making it ideal for complex technical documents where positional context matters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}