What Scripts Run the EXTRACTOR Phase of Book-to-Skill: Complete Pipeline Guide

The EXTRACTOR phase of Book-to-Skill is driven by seven primary scripts: an entry-point wrapper (scripts/extract.py), a CLI orchestrator (book_to_skill/cli.py), format-specific parsers in book_to_skill/parsers/, and a text-sanitization engine (book_to_skill/sanitize.py), supported by configuration and utility modules.

The EXTRACTOR phase transforms source documents—PDF, EPUB, DOCX, RTF, HTML, and plain text—into normalized, searchable content for downstream skill generation. In the virgiliojr94/book-to-skill repository, this phase follows a deterministic pipeline architecture that separates parsing logic from CLI concerns.

Entry Point and CLI Orchestration

The extraction workflow begins with two coordination layers that handle I/O setup and argument parsing.

scripts/extract.py – The UTF-8 Enforcement Wrapper

Located at scripts/extract.py, this module serves as the thin CLI entry point. It forces UTF-8 encoding for standard streams and delegates execution to the package’s internal API. Invoke it directly with:

python -m scripts.extract path/to/book.pdf

book_to_skill/cli.py – Core Driver Logic

The book_to_skill/cli.py file contains the main() function that parses command-line arguments, determines the appropriate document parser based on file extension or user hints, and orchestrates the full extraction flow. It reads the input file, invokes the selected parser, passes the raw string to the sanitizer, and writes the final output to disk.

Document Parsers in book_to_skill/parsers/

Each parser implements a consistent parse(path: str) -> str interface. The CLI dynamically selects the correct implementation from the book_to_skill/parsers/ directory based on the input format.

• book_to_skill/parsers/pdf.py – Extracts text from PDF files using pdfminer or pymupdf.
• book_to_skill/parsers/epub.py – Walks EPUB packages and concatenates XHTML chapter content.
• book_to_skill/parsers/docx.py – Reads Microsoft Word documents via python-docx.
• book_to_skill/parsers/html.py – Strips markup and isolates primary content from HTML files.
• book_to_skill/parsers/rtf.py – Handles Rich Text Format decoding.
• book_to_skill/parsers/text.py – Passes through raw .txt files unchanged.
• book_to_skill/parsers/calibre.py – Provides optional Calibre-based conversion for obscure or legacy formats.

Text Sanitization and Utilities

After raw extraction, the pipeline normalizes content through dedicated processing modules.

book_to_skill/sanitize.py – Post-Extraction Cleaning

The sanitize_text() function in book_to_skill/sanitize.py normalizes Unicode forms, removes problematic control characters, and strips boilerplate artifacts that commonly appear in extracted text.

book_to_skill/utils.py – Shared Infrastructure

book_to_skill/utils.py supplies generic helpers for path manipulation and logging used across the CLI and parser modules.

Configuration and Error Handling

• book_to_skill/config.py – Stores default options including output directory paths and language settings.
• book_to_skill/exceptions.py – Defines a unified exception hierarchy for the extraction step, allowing the CLI to present typed, user-friendly error messages.

How to Run the Extraction Phase

Command-Line Execution

Extract a PDF and write cleaned text to the default output folder:

python -m scripts.extract path/to/book.pdf --out-dir ./extracted/

Programmatic API Usage

Invoke the CLI logic directly from Python by passing a list of arguments to main():

from book_to_skill.cli import main

args = ["path/to/book.epub", "--out-dir", "out"]
main(args)  # Returns None; output written to disk

Direct Parser Access

Bypass the CLI entirely to work with extracted text in memory:

from book_to_skill.parsers.pdf import PDFParser
from book_to_skill.sanitize import sanitize_text

raw = PDFParser().parse("my_book.pdf")
clean = sanitize_text(raw)
print(clean[:500])  # Preview first 500 characters

Summary

• The EXTRACTOR phase is initiated via scripts/extract.py, which ensures UTF-8 I/O before calling the package CLI.
• book_to_skill/cli.py coordinates the workflow, routing files to the correct parser in book_to_skill/parsers/.
• Seven format-specific parsers implement the parse(path: str) -> str interface for PDF, EPUB, DOCX, HTML, RTF, plain text, and Calibre-compatible formats.
• book_to_skill/sanitize.py provides the sanitize_text() function to normalize Unicode and remove control characters.
• Configuration defaults live in book_to_skill/config.py, while book_to_skill/exceptions.py supplies the error taxonomy.

Frequently Asked Questions

What file formats does the EXTRACTOR phase support?

The EXTRACTOR phase supports PDF, EPUB, Microsoft Word (DOCX), RTF, HTML, plain text, and any format convertible via Calibre. Each format has a dedicated parser module in book_to_skill/parsers/ that implements the standard parse() interface.

How does the CLI select the correct parser for a document?

The main() function in book_to_skill/cli.py inspects the file extension of the input path. It maps extensions to parser classes—for example, .pdf routes to PDFParser—and instantiates the appropriate class to extract text.

Can I extract text programmatically without writing to disk?

Yes. Import the specific parser from book_to_skill.parsers (such as PDFParser or EPUBParser) and call its parse() method directly. Pass the resulting string to sanitize_text() from book_to_skill/sanitize.py to replicate the full pipeline in memory without filesystem I/O.

Where is the extracted text sanitized during the pipeline?

Sanitization occurs in book_to_skill/sanitize.py. After the CLI receives raw text from a parser, it passes the string through sanitize_text() to normalize Unicode, strip control characters, and remove extraction artifacts before writing the final output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}