# Book-to-Skill Extraction Pipeline: How Files Are Processed Across 6 Document Formats

> Discover the 5 steps in the book-to-skill extraction pipeline. Learn how files are processed across 6 document formats from detection to payload assembly.

- Repository: [Virgilio Junior/book-to-skill](https://github.com/virgiliojr94/book-to-skill)
- Tags: internals
- Published: 2026-08-25

---

**The book-to-skill extraction pipeline follows five deterministic steps: file-type detection, format-specific text extraction, optional PDF post-processing, universal text sanitization, and Skill payload assembly.**

The **book-to-skill** package transforms source documents into clean, plain-text representations suitable for Skill generation. Written by virgiliojr94, this open-source tool handles multiple document formats through a modular architecture where each format follows the same high-level pipeline with specialized extraction logic.

## Step 1: File-Type Detection and Parser Dispatch

The CLI entry point in [`book_to_skill/cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/cli.py) inspects the input file's extension and routes to the appropriate parser module. Supported extensions include `.pdf`, `.html`, `.epub`, `.docx`, `.rtf`, and `.txt`.

This dispatch mechanism ensures that format-specific complexity is isolated within dedicated parser modules while maintaining a consistent interface across all document types.

## Step 2: Format-Specific Text Extraction

Each parser implements an `extract_*` function that returns a Unicode string or `None` on failure. The extraction strategies vary significantly by format:

### PDF Extraction ([`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py))

The PDF parser implements a **fallback cascade** that tries multiple extraction engines in order of preference:

1. **pdftotext** – external command-line tool (fastest, layout-aware)
2. **pypdf** – pure-Python library
3. **pdfminer** – robust but slower alternative
4. **docling** – layout-aware deep learning parser

```python
from book_to_skill.parsers.pdf import (
    extract_with_pdftotext,
    extract_with_pypdf,
    extract_with_pdfminer,
    extract_with_docling
)

pdf_path = "my_book.pdf"

raw = extract_with_pdftotext(pdf_path) \
      or extract_with_pypdf(pdf_path) \
      or extract_with_pdfminer(pdf_path) \
      or extract_with_docling(pdf_path)

```

### HTML Extraction ([`book_to_skill/parsers/html.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/html.py))

Parses documents with **BeautifulSoup**, removes boilerplate markup, and concatenates visible text content.

### EPUB Extraction ([`book_to_skill/parsers/epub.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/epub.py))

Unpacks the ZIP archive, walks the manifest, and extracts text from each XHTML chapter sequentially.

### DOCX Extraction ([`book_to_skill/parsers/docx.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/docx.py))

Uses `python-docx` to read paragraph elements from Word documents.

```python
from book_to_skill.parsers.docx import extract_with_docx

docx_path = "chapter.docx"
raw = extract_with_docx(docx_path)

```

### RTF Extraction ([`book_to_skill/parsers/rtf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/rtf.py))

Streams RTF content through a lightweight native parser.

### Plain Text Extraction ([`book_to_skill/parsers/text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/text.py))

Handles byte-order-mark detection and legacy encoding decoding via `read_text_file`.

## Step 3: PDF-Specific Post-Processing

When **pdftotext** is used with the `-layout` flag, its output receives additional cleanup through `clean_pdftotext` in [`book_to_skill/parsers/pdf.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/pdf.py). This function:

- Removes repeated headers and footers
- Drops standalone page numbers
- De-hyphenates words split across line breaks

This post-processing step applies only to PDF extraction and operates before universal sanitization.

## Step 4: Universal Text Sanitization

All extracted text—regardless of source format—passes through [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py). The `sanitize_text` function and helpers like `remove_bidi_controls` perform:

- Stripping of problematic Unicode control characters
- Whitespace normalization
- UTF-8 safety guarantees

```python
from book_to_skill.sanitize import sanitize_text

clean = sanitize_text(raw) if raw else ""
print(clean[:500])  # preview first 500 characters

```

## Step 5: Skill Payload Assembly

The sanitized text is combined with metadata (title, author, source-type) and written to [`SKILL.md`](https://github.com/virgiliojr94/book-to-skill/blob/main/SKILL.md) by the generator component. This final step transitions from deterministic extraction to spec-driven Skill generation.

## Complete CLI Example

Extract any supported format from the command line:

```bash
python -m book_to_skill --input my_book.pdf --output my_skill.md

```

A subprocess-friendly wrapper is also available in [`scripts/extract.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/scripts/extract.py).

## Summary

- **File-type dispatch** in [`cli.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/cli.py) routes to specialized parsers based on extension
- **Format-specific extraction** uses optimal strategies: pdftotext/pypdf/pdfminer/docling for PDF, BeautifulSoup for HTML, ZIP manifest walking for EPUB, python-docx for DOCX, native streaming for RTF, and BOM-aware reading for plain text
- **PDF post-processing** via `clean_pdftotext` removes artifacts from layout-preserving extraction
- **Universal sanitization** through `sanitize_text` ensures clean UTF-8 output
- **Skill assembly** combines cleaned text with metadata for final output

## Frequently Asked Questions

### What happens if PDF extraction fails with pdftotext?

The pipeline automatically falls back through **pypdf**, **pdfminer**, and finally **docling** until successful extraction occurs. Each fallback trades speed for robustness or layout awareness.

### Does the pipeline preserve document structure?

Basic structural preservation varies by format. The **pdftotext -layout** mode and **docling** parser maintain spatial relationships, while HTML and EPUB extractors preserve reading order from the markup. The final Skill markdown is intentionally flattened to plain text.

### Can I extract text programmatically without using the CLI?

Yes. Import directly from the parser modules and [`sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/sanitize.py) as shown in the code examples. This bypasses the CLI's file-type detection, so you must explicitly call the correct `extract_*` function for your document format.

### How does the pipeline handle encoding issues?

The [`book_to_skill/parsers/text.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/parsers/text.py) module detects byte-order-marks and attempts legacy encoding fallback. All extracted content ultimately passes through `sanitize_text` in [`book_to_skill/sanitize.py`](https://github.com/virgiliojr94/book-to-skill/blob/main/book_to_skill/sanitize.py) to guarantee valid UTF-8 output.