How to Extend book-to-skill by Adding a New Format Parser: A Step-by-Step Guide

To extend book-to-skill with a new format parser, create a Python module in book_to_skill/parsers/ that implements an extraction function, register the file extension in utils.py, expose the module in parsers/__init__.py, and add tests to verify integration.

book-to-skill is an open-source document processing tool that extracts raw text from multiple source formats through a clean, modular parser architecture. The parser layer lives in book_to_skill/parsers/ and follows a simple contract: each module exposes a function that receives a file path and returns extracted text or None on failure. Understanding how to extend this system lets developers add support for any custom document format without modifying core business logic.

How the Parser Architecture Works

The extraction pipeline centers on extract_single_file in book_to_skill/utils.py. This dispatcher inspects file extensions, looks up the corresponding parser in a registry dictionary, and delegates extraction to the appropriate module. This design keeps format-specific code isolated and makes adding new parsers a matter of implementing three touch-points: the parser module, the registry entry, and package exports.

Key reference implementations demonstrate the pattern:

• book_to_skill/parsers/text.py — Minimal example with BOM handling and encoding detection
• book_to_skill/parsers/pdf.py — Multiple fallback extraction strategies
• book_to_skill/parsers/html.py — HTML-specific cleaning and normalization

Step 1: Create the Parser Module

Create a new .py file in book_to_skill/parsers/ following the established function signature. The function must accept a path string and return str | None.

# book_to_skill/parsers/myformat.py

from __future__ import annotations
import sys
from pathlib import Path

def extract_myformat(path: str) -> str | None:
    """Extract raw text from a *.myfmt file.

    Returns the extracted string or ``None`` on failure.
    """
    try:
        data = Path(path).read_bytes()
    except Exception as e:
        print(f"  [warn] extract_myformat failed: {type(e).__name__}: {e}",
              file=sys.stderr)
        return None

    # Example: binary format starts with a custom BOM.

    if data.startswith(b"\x01\x02\x03\x04"):
        try:
            return data.decode("utf-8")
        except UnicodeDecodeError:
            pass

    # Fallback – try common encodings.

    for enc in ("utf-8", "cp1252", "latin-1"):
        try:
            return data.decode(enc)
        except UnicodeDecodeError:
            continue
    return None

Design recommendations:

• Handle binary and text formats gracefully
• Return None rather than raising exceptions on unrecoverable errors
• Print warnings to stderr for debugging without breaking downstream processing
• Provide encoding fallbacks for text-based formats

Step 2: Register in the Parser Registry

Open book_to_skill/utils.py and locate the PARSER_BY_EXT dictionary. Import your new module and add an entry mapping the file extension to your extraction function.

# book_to_skill/utils.py

from .parsers import myformat

PARSER_BY_EXT = {
    ".txt": parsers.text.read_text_file,
    ".pdf": parsers.pdf.extract_pdf,
    ".html": parsers.html.extract_html_file,
    ".epub": parsers.epub.extract_epub,
    ".docx": parsers.docx.extract_docx,
    ".rtf": parsers.rtf.extract_rtf,
    ".myfmt": myformat.extract_myformat,   # ← new entry

}

The registry is the only place where your new parser connects to the core extraction pipeline. This single-point registration keeps dependencies explicit and reviewable.

Step 3: Expose the Module in Package Exports

Update book_to_skill/parsers/__init__.py to include your module in the package's public interface:

# book_to_skill/parsers/__init__.py

from . import text, pdf, html, epub, docx, rtf, calibre, myformat

This import ensures that utils.py can resolve the module when it performs its lookup during extraction.

Step 4: Update CLI Help Text (Optional)

In book_to_skill/cli.py, locate where supported formats are listed for user help. Add your new extension to maintain accurate documentation:

# Example location in cli.py

SUPPORTED_FORMATS = [".txt", ".pdf", ".html", ".epub", ".docx", ".rtf", ".myfmt"]

This step improves discoverability and prevents user confusion about which formats the tool accepts.

Step 5: Write Integration Tests

Add a test file under tests/ that exercises your parser through the full extraction pipeline:

# tests/test_myformat_parser.py

def test_myformat_extraction(tmp_path):
    sample = tmp_path / "sample.myfmt"
    sample.write_bytes(b"\x01\x02\x03\x04Hello world!")
    from book_to_skill.utils import extract_single_file
    result = extract_single_file(
        sample,
        extraction_mode="myfmt",
        install_mode="none"
    )
    assert "Hello world!" in result["text"]

Run the test suite to verify your implementation:

pytest -q tests/test_myformat_parser.py

Step 6: Validate with Repository Checks

Before submitting changes, ensure code quality and compatibility:

ruff check .          # Static analysis

pytest -q             # Full test suite

These checks confirm that your new parser integrates cleanly without regressions.

Key Files for Parser Development

File	Purpose
book_to_skill/parsers/	Directory containing all format-specific extractors
book_to_skill/parsers/text.py	Minimal reference implementation
book_to_skill/parsers/pdf.py	Complex parser with multiple strategies
book_to_skill/utils.py	Core dispatcher and PARSER_BY_EXT registry
book_to_skill/cli.py	Command-line interface and help text
tests/	Test fixtures and validation suite

Summary

• Create a parser module in book_to_skill/parsers/ with an extract_<fmt>(path: str) -> str | None function
• Register the extension-to-function mapping in utils.py's PARSER_BY_EXT dictionary
• Export the module in parsers/__init__.py for package visibility
• Document the new format in cli.py help text
• Test with pytest using the extract_single_file utility to verify end-to-end integration
• Validate with ruff and the full test suite before contribution

The parser layer's deterministic, isolated design guarantees that adding a new format parser touches only the registry and optional CLI documentation—no core logic changes required.

Frequently Asked Questions

What function signature must a custom parser implement?

A custom parser must implement def extract_<fmt>(path: str) -> str | None: where path is the absolute file path and the return value is the extracted text string or None if extraction fails. This signature matches all built-in parsers in book_to_skill/parsers/ and ensures compatibility with the extract_single_file dispatcher.

Can a parser return metadata alongside text?

The base contract returns only text, but you can extend the return structure if downstream code is updated accordingly. The current extract_single_file implementation in utils.py expects a string return and packages it into a result dictionary. For metadata-rich formats, consider returning a structured string or modifying the dispatcher to handle dictionary returns.

How does book-to-skill handle parser failures gracefully?

Each parser catches exceptions internally, prints warnings to stderr, and returns None on failure. The extract_single_file function in utils.py handles None returns by propagating empty results or fallback behavior, ensuring that one parser failure does not crash the entire extraction pipeline.

Is it possible to override an existing parser for a format?

Yes. Simply register your replacement function in PARSER_BY_EXT using the same extension key. The dictionary lookup uses the last-assigned value, so your custom parser will supersede the built-in implementation. This technique enables patching or enhancing extraction for problematic formats without forking the repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}