Book-to-Skill Extraction Pipeline: How Files Are Processed Across 6 Document Formats

The book-to-skill extraction pipeline follows five deterministic steps: file-type detection, format-specific text extraction, optional PDF post-processing, universal text sanitization, and Skill payload assembly.

The book-to-skill package transforms source documents into clean, plain-text representations suitable for Skill generation. Written by virgiliojr94, this open-source tool handles multiple document formats through a modular architecture where each format follows the same high-level pipeline with specialized extraction logic.

Step 1: File-Type Detection and Parser Dispatch

The CLI entry point in book_to_skill/cli.py inspects the input file's extension and routes to the appropriate parser module. Supported extensions include .pdf, .html, .epub, .docx, .rtf, and .txt.

This dispatch mechanism ensures that format-specific complexity is isolated within dedicated parser modules while maintaining a consistent interface across all document types.

Step 2: Format-Specific Text Extraction

Each parser implements an extract_* function that returns a Unicode string or None on failure. The extraction strategies vary significantly by format:

PDF Extraction (book_to_skill/parsers/pdf.py)

The PDF parser implements a fallback cascade that tries multiple extraction engines in order of preference:

1. pdftotext – external command-line tool (fastest, layout-aware)
2. pypdf – pure-Python library
3. pdfminer – robust but slower alternative
4. docling – layout-aware deep learning parser

from book_to_skill.parsers.pdf import (
    extract_with_pdftotext,
    extract_with_pypdf,
    extract_with_pdfminer,
    extract_with_docling
)

pdf_path = "my_book.pdf"

raw = extract_with_pdftotext(pdf_path) \
      or extract_with_pypdf(pdf_path) \
      or extract_with_pdfminer(pdf_path) \
      or extract_with_docling(pdf_path)

HTML Extraction (book_to_skill/parsers/html.py)

Parses documents with BeautifulSoup, removes boilerplate markup, and concatenates visible text content.

EPUB Extraction (book_to_skill/parsers/epub.py)

Unpacks the ZIP archive, walks the manifest, and extracts text from each XHTML chapter sequentially.

DOCX Extraction (book_to_skill/parsers/docx.py)

Uses python-docx to read paragraph elements from Word documents.

from book_to_skill.parsers.docx import extract_with_docx

docx_path = "chapter.docx"
raw = extract_with_docx(docx_path)

RTF Extraction (book_to_skill/parsers/rtf.py)

Streams RTF content through a lightweight native parser.

Plain Text Extraction (book_to_skill/parsers/text.py)

Handles byte-order-mark detection and legacy encoding decoding via read_text_file.

Step 3: PDF-Specific Post-Processing

When pdftotext is used with the -layout flag, its output receives additional cleanup through clean_pdftotext in book_to_skill/parsers/pdf.py. This function:

• Removes repeated headers and footers
• Drops standalone page numbers
• De-hyphenates words split across line breaks

This post-processing step applies only to PDF extraction and operates before universal sanitization.

Step 4: Universal Text Sanitization

All extracted text—regardless of source format—passes through book_to_skill/sanitize.py. The sanitize_text function and helpers like remove_bidi_controls perform:

• Stripping of problematic Unicode control characters
• Whitespace normalization
• UTF-8 safety guarantees

from book_to_skill.sanitize import sanitize_text

clean = sanitize_text(raw) if raw else ""
print(clean[:500])  # preview first 500 characters

Step 5: Skill Payload Assembly

The sanitized text is combined with metadata (title, author, source-type) and written to SKILL.md by the generator component. This final step transitions from deterministic extraction to spec-driven Skill generation.

Complete CLI Example

Extract any supported format from the command line:

python -m book_to_skill --input my_book.pdf --output my_skill.md

A subprocess-friendly wrapper is also available in scripts/extract.py.

Summary

• File-type dispatch in cli.py routes to specialized parsers based on extension
• Format-specific extraction uses optimal strategies: pdftotext/pypdf/pdfminer/docling for PDF, BeautifulSoup for HTML, ZIP manifest walking for EPUB, python-docx for DOCX, native streaming for RTF, and BOM-aware reading for plain text
• PDF post-processing via clean_pdftotext removes artifacts from layout-preserving extraction
• Universal sanitization through sanitize_text ensures clean UTF-8 output
• Skill assembly combines cleaned text with metadata for final output

Frequently Asked Questions

What happens if PDF extraction fails with pdftotext?

The pipeline automatically falls back through pypdf, pdfminer, and finally docling until successful extraction occurs. Each fallback trades speed for robustness or layout awareness.

Does the pipeline preserve document structure?

Basic structural preservation varies by format. The pdftotext -layout mode and docling parser maintain spatial relationships, while HTML and EPUB extractors preserve reading order from the markup. The final Skill markdown is intentionally flattened to plain text.

Can I extract text programmatically without using the CLI?

Yes. Import directly from the parser modules and sanitize.py as shown in the code examples. This bypasses the CLI's file-type detection, so you must explicitly call the correct extract_* function for your document format.

How does the pipeline handle encoding issues?

The book_to_skill/parsers/text.py module detects byte-order-marks and attempts legacy encoding fallback. All extracted content ultimately passes through sanitize_text in book_to_skill/sanitize.py to guarantee valid UTF-8 output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}