How the book-to-skill Extractor Component Works: A Complete Technical Guide

The book-to-skill extractor transforms documents (PDF, EPUB, DOCX, RTF, HTML, and plain text) into clean, sanitized plain text through a registry-based parser pipeline.

The book-to-skill project is an open-source tool that converts book content into skill-generation training data. At its core sits the extractor component, a modular pipeline that handles format detection, content extraction, and text sanitization. This guide examines the complete extraction flow from source code to output.

Entry Point and CLI Orchestration

The extraction journey begins at scripts/extract.py, a minimal wrapper that ensures the repository root is present on sys.path before delegating to the package's CLI entry point book_to_skill.cli.main.

# scripts/extract.py

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent.parent))

from book_to_skill.cli import main

if __name__ == "__main__":
    main()

The book_to_skill/cli.py module handles argument parsing and parser selection. It inspects the input file's extension or MIME type, then routes to the appropriate extraction logic in book_to_skill/utils.py.

Parser Registry Architecture

The extractor uses a centralized registry pattern defined in book_to_skill/utils.py. This dictionary maps file extensions to concrete parser classes:

PARSER_REGISTRY = {
    ".pdf":  PDFParser,
    ".epub": EpubParser,
    ".docx": DocxParser,
    ".rtf":  RtfParser,
    ".html": HtmlParser,
    ".txt":  TextParser,
}

This design allows easy extension — adding support for a new format requires only implementing a parser class and registering it here.

Format-Specific Parser Implementations

Each parser module in book_to_skill/parsers/ implements a consistent interface: extract(path: str) -> str. Here's how each format is handled:

Plain Text Parser

book_to_skill/parsers/text.py handles encoding detection with BOM awareness and intelligent fallback chains:

def extract(path: str) -> str:
    with open(path, "rb") as f:
        raw = f.read()
    
    # Check for BOM markers (UTF-8, UTF-16, UTF-32)

    encoding = _detect_bom(raw)
    if encoding:
        return raw.decode(encoding, errors="replace")
    
    # Fallback to common encodings

    for enc in ("utf-8", "cp1252", "latin-1"):
        try:
            return raw.decode(enc)
        except UnicodeDecodeError:
            continue
    
    return raw.decode("utf-8", errors="replace")  # Last resort

PDF Parser

book_to_skill/parsers/pdf.py uses pdfminer.six to extract text page-by-page while preserving logical paragraph structure.

EPUB Parser

book_to_skill/parsers/epub.py opens the ZIP archive, walks the HTML spine, and concatenates cleaned text from internal documents.

DOCX Parser

book_to_skill/parsers/docx.py leverages python-docx to extract paragraphs in document order.

RTF and HTML Parsers

• RTF (book_to_skill/parsers/rtf.py): Parses RTF control words and returns raw character content
• HTML (book_to_skill/parsers/html.py): Uses BeautifulSoup to strip tags, preserve visible text, and collapse excessive whitespace

Text Sanitization Pipeline

After extraction, book_to_skill/sanitize.py performs critical cleaning operations:

1. Control character removal — Strips zero-width spaces, bidirectional markers, and other invisible Unicode characters
2. Whitespace normalization — Standardizes line endings and collapses multiple spaces
3. Annotation stripping — Removes proprietary markers that could leak copyrighted material or break downstream processing

This stage ensures the extracted text is safe for the skill-generation engine.

Usage Examples

Command-Line Extraction

python -m book_to_skill extract path/to/book.pdf > cleaned.txt

Programmatic API

from book_to_skill.utils import extract_text

clean_text = extract_text("my_book.epub")
print(clean_text[:500])  # Preview first 500 characters

Direct Parser Access

from book_to_skill.parsers.pdf import extract as pdf_extract
from book_to_skill.sanitize import sanitize_text

raw = pdf_extract("my_book.pdf")
clean = sanitize_text(raw)  # Manual sanitization when needed

Summary

• The book-to-skill extractor follows a deterministic pipeline: wrapper → CLI → registry → parser → sanitizer
• Registry pattern in book_to_skill/utils.py enables clean format extension
• Six supported formats: PDF, EPUB, DOCX, RTF, HTML, and plain text with intelligent encoding detection
• Mandatory sanitization removes control characters and normalizes whitespace before output
• Dual interfaces support both CLI scripting and Python library integration

Frequently Asked Questions

What file formats does the book-to-skill extractor support?

The extractor supports PDF, EPUB, DOCX, RTF, HTML, and plain text files. Each format has a dedicated parser class registered in book_to_skill/utils.py. You can check the PARSER_REGISTRY dictionary to see the current mapping of extensions to parser implementations.

How does the extractor handle files with unknown or legacy encodings?

The plain text parser in book_to_skill/parsers/text.py implements a three-tier fallback strategy. First, it detects UTF-8/UTF-16/UTF-32 BOM markers. If none are found, it attempts decoding in order: utf-8 → cp1252 → latin-1. As a last resort, it uses utf-8 with error replacement to guarantee valid output.

Can I use the extractor without installing the full book-to-skill package?

Yes. The modular parser design allows direct import of individual parsers. For example, from book_to_skill.parsers.docx import extract gives you format-specific extraction without CLI overhead. However, you'll need to run sanitize_text() manually if you bypass the standard pipeline.

Where should I add a new format parser?

Create a new module in book_to_skill/parsers/ implementing extract(path: str) -> str, then register the class in PARSER_REGISTRY within book_to_skill/utils.py. The CLI and utility functions will automatically recognize the new format without additional changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}