# How the book-to-skill Extractor Component Works: A Complete Technical Guide

> Discover how the book-to-skill extractor component works. This technical guide explains how documents transform into clean plain text using a registry-based parser pipeline. Learn more.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: deep-dive
- Published: 2026-08-27

---

**The book-to-skill extractor transforms documents (PDF, EPUB, DOCX, RTF, HTML, and plain text) into clean, sanitized plain text through a registry-based parser pipeline.**

The **book-to-skill** project is an open-source tool that converts book content into skill-generation training data. At its core sits the **extractor component**, a modular pipeline that handles format detection, content extraction, and text sanitization. This guide examines the complete extraction flow from source code to output.

## Entry Point and CLI Orchestration

The extraction journey begins at [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py), a minimal wrapper that ensures the repository root is present on `sys.path` before delegating to the package's CLI entry point `book_to_skill.cli.main`.

```python

# scripts/extract.py

import sys
from pathlib import Path

sys.path.insert(0, str(Path(__file__).parent.parent))

from book_to_skill.cli import main

if __name__ == "__main__":
    main()

```

The [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) module handles argument parsing and parser selection. It inspects the input file's extension or MIME type, then routes to the appropriate extraction logic in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py).

## Parser Registry Architecture

The extractor uses a centralized registry pattern defined in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). This dictionary maps file extensions to concrete parser classes:

```python
PARSER_REGISTRY = {
    ".pdf":  PDFParser,
    ".epub": EpubParser,
    ".docx": DocxParser,
    ".rtf":  RtfParser,
    ".html": HtmlParser,
    ".txt":  TextParser,
}

```

This design allows **easy extension** — adding support for a new format requires only implementing a parser class and registering it here.

## Format-Specific Parser Implementations

Each parser module in `book_to_skill/parsers/` implements a consistent interface: `extract(path: str) -> str`. Here's how each format is handled:

### Plain Text Parser

[`book_to_skill/parsers/text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/text.py) handles encoding detection with BOM awareness and intelligent fallback chains:

```python
def extract(path: str) -> str:
    with open(path, "rb") as f:
        raw = f.read()
    
    # Check for BOM markers (UTF-8, UTF-16, UTF-32)

    encoding = _detect_bom(raw)
    if encoding:
        return raw.decode(encoding, errors="replace")
    
    # Fallback to common encodings

    for enc in ("utf-8", "cp1252", "latin-1"):
        try:
            return raw.decode(enc)
        except UnicodeDecodeError:
            continue
    
    return raw.decode("utf-8", errors="replace")  # Last resort

```

### PDF Parser

[`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py) uses `pdfminer.six` to extract text page-by-page while preserving logical paragraph structure.

### EPUB Parser

[`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py) opens the ZIP archive, walks the HTML spine, and concatenates cleaned text from internal documents.

### DOCX Parser

[`book_to_skill/parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/docx.py) leverages `python-docx` to extract paragraphs in document order.

### RTF and HTML Parsers

- **RTF** ([`book_to_skill/parsers/rtf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/rtf.py)): Parses RTF control words and returns raw character content
- **HTML** ([`book_to_skill/parsers/html.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/html.py)): Uses BeautifulSoup to strip tags, preserve visible text, and collapse excessive whitespace

## Text Sanitization Pipeline

After extraction, [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) performs **critical cleaning operations**:

1. **Control character removal** — Strips zero-width spaces, bidirectional markers, and other invisible Unicode characters
2. **Whitespace normalization** — Standardizes line endings and collapses multiple spaces
3. **Annotation stripping** — Removes proprietary markers that could leak copyrighted material or break downstream processing

This stage ensures the extracted text is safe for the skill-generation engine.

## Usage Examples

### Command-Line Extraction

```bash
python -m book_to_skill extract path/to/book.pdf > cleaned.txt

```

### Programmatic API

```python
from book_to_skill.utils import extract_text

clean_text = extract_text("my_book.epub")
print(clean_text[:500])  # Preview first 500 characters

```

### Direct Parser Access

```python
from book_to_skill.parsers.pdf import extract as pdf_extract
from book_to_skill.sanitize import sanitize_text

raw = pdf_extract("my_book.pdf")
clean = sanitize_text(raw)  # Manual sanitization when needed

```

## Summary

- The **book-to-skill extractor** follows a deterministic pipeline: wrapper → CLI → registry → parser → sanitizer
- **Registry pattern** in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py) enables clean format extension
- **Six supported formats**: PDF, EPUB, DOCX, RTF, HTML, and plain text with intelligent encoding detection
- **Mandatory sanitization** removes control characters and normalizes whitespace before output
- **Dual interfaces** support both CLI scripting and Python library integration

## Frequently Asked Questions

### What file formats does the book-to-skill extractor support?

The extractor supports **PDF, EPUB, DOCX, RTF, HTML, and plain text** files. Each format has a dedicated parser class registered in [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). You can check the `PARSER_REGISTRY` dictionary to see the current mapping of extensions to parser implementations.

### How does the extractor handle files with unknown or legacy encodings?

The plain text parser in [`book_to_skill/parsers/text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/text.py) implements a **three-tier fallback strategy**. First, it detects UTF-8/UTF-16/UTF-32 BOM markers. If none are found, it attempts decoding in order: `utf-8` → `cp1252` → `latin-1`. As a last resort, it uses `utf-8` with error replacement to guarantee valid output.

### Can I use the extractor without installing the full book-to-skill package?

Yes. The modular parser design allows **direct import of individual parsers**. For example, `from book_to_skill.parsers.docx import extract` gives you format-specific extraction without CLI overhead. However, you'll need to run `sanitize_text()` manually if you bypass the standard pipeline.

### Where should I add a new format parser?

Create a new module in `book_to_skill/parsers/` implementing `extract(path: str) -> str`, then register the class in `PARSER_REGISTRY` within [`book_to_skill/utils.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/utils.py). The CLI and utility functions will automatically recognize the new format without additional changes.