Can book-to-skill Preserve Markdown Tables and Code Blocks from PDFs?

No, the standard book-to-skill pipeline converts PDFs to plain text using pdftotext, which flattens visual formatting including Markdown tables and fenced code blocks.

The book-to-skill library extracts raw textual content from PDF documents to generate structured skill manifests, but it does not retain the original Markdown syntax or table layouts. Unless you implement custom post-processing logic, any tabular data or code blocks present in the source PDF will arrive as unformatted paragraphs in the final output.

How book-to-skill Processes PDF Content

The extraction workflow treats PDFs as flat text streams rather than structured documents. In book_to_skill/parsers/pdf.py, the parser invokes the pdftotext utility to convert each page into a string of paragraphs. This text then flows through book_to_skill/sanitize.py for normalization—removing bidirectional control characters and standardizing line breaks—before reaching the skill generation logic orchestrated by book_to_skill/cli.py.

Because pdftotext extracts visible characters without preserving the semantic markup of the original document, Markdown delimiters such as pipe characters (|) for tables or triple backticks (`) for code fences are not explicitly tagged. If the PDF rendered a table using visual spacing or monospaced fonts, the extraction tool sees only the raw words and whitespace, not the structural boundaries.

Why Markdown Structure Is Lost

The core limitation lies in the PDF-to-text conversion step. When pdftotext processes a PDF, it strips away:

• Column alignment and borders that visually define tables
• Font metadata that might indicate code blocks (e.g., monospaced typefaces)
• Indentation hierarchy that distinguishes code from body text

The resulting string contains the words from your tables and code snippets, but without the surrounding Markdown syntax needed to reconstruct them. For example, a rendered table appears as space-separated values, and a code block becomes a series of indented lines indistinguishable from regular text until post-processing occurs.

Implementing Custom Post-Processing

To preserve Markdown tables and code blocks from PDFs using book-to-skill, you must intercept the extracted text between the parser and sanitization stages. You can modify scripts/extract.py or extend the pipeline in book_to_skill/cli.py to apply pattern detection heuristics.

Detecting and Reconstructing Markdown Tables

After PdfParser.parse() returns the raw string, scan for table-like patterns—rows containing multiple pipe characters (|) or columns separated by three or more spaces—and reformat them into valid Markdown table syntax with header separators.

def reconstruct_markdown_tables(text: str) -> str:
    lines = text.splitlines()
    table_lines = []
    output = []
    
    for line in lines:
        # Detect pipe-separated rows or space-aligned columns

        if "|" in line or "   " in line and len(line.split()) >= 2:
            table_lines.append(line)
        else:
            if table_lines:
                # Convert accumulated lines to Markdown table

                header = table_lines[0].replace("  ", "|").strip()
                separator = "|".join(["---"] * len(header.split("|")))
                output.extend([header, separator] + table_lines[1:])
                table_lines = []
            output.append(line)
    
    return "\n".join(output)

Recovering Fenced Code Blocks

Identify code sections by looking for consistent indentation (four or more spaces) or monospaced font patterns, then wrap them with triple backticks. Insert this logic after the PDF extraction but before the text reaches book_to_skill/sanitize.py.

import re

def fence_code_blocks(text: str) -> str:
    # Match blocks of 3+ consecutive lines starting with 4+ spaces

    pattern = r"(?:^ {4,}.+\n){3,}"
    matches = list(re.finditer(pattern, text, re.MULTILINE))
    
    # Replace from end to avoid offset issues

    for match in reversed(matches):
        block = match.group(0)
        # Remove leading indentation and wrap

        cleaned = "\n".join(line[4:] for line in block.rstrip().splitlines())
        fenced = f"```\n{cleaned}\n```\n"
        text = text[:match.start()] + fenced + text[match.end():]
    
    return text

Key Source Files in book-to-skill

Understanding these specific implementation files helps when planning modifications:

• book_to_skill/parsers/pdf.py — Contains the pdftotext invocation logic that converts PDF pages to strings.
• scripts/extract.py — Entry point script that initializes the parser and handles raw extraction workflows.
• book_to_skill/sanitize.py — Applies generic text cleaning operations but does not restore structural formatting.
• book_to_skill/cli.py — Orchestrates the complete pipeline from PDF ingestion to skill manifest generation.

Summary

• book-to-skill uses pdftotext utility in book_to_skill/parsers/pdf.py to extract PDF content as plain text.
• The standard pipeline flattens visual formatting, discarding Markdown table syntax and code fence delimiters.
• Extracted tables appear as space-separated text; code blocks appear as indented paragraphs without language identifiers.
• You must implement custom post-processing between the parser and sanitization stages to reconstruct Markdown structures.
• Modify scripts/extract.py or extend book_to_skill/cli.py with pattern detection logic to recover tables and code blocks.

Frequently Asked Questions

Does book-to-skill support native PDF table extraction?

No, the library does not include native support for preserving table structures. According to the source code in book_to_skill/parsers/pdf.py, the tool relies entirely on pdftotext, which outputs linear text without cell boundaries or column metadata. You need custom logic to detect table patterns and regenerate Markdown pipe syntax.

Can book-to-skill distinguish code blocks from regular text in PDFs?

By default, no. The extraction process in book_to_skill/parsers/pdf.py treats all content as uniform text streams. While the PDF may contain font metadata indicating monospaced code sections, pdftotext does not expose this information. You must implement heuristics—such as detecting consistent indentation or specific keywords—to identify and fence code blocks during post-processing.

Where should I add custom formatting logic in the book-to-skill pipeline?

Insert your reconstruction functions between the PDF parser and the sanitization module. Specifically, modify the workflow in scripts/extract.py or create a preprocessor that runs after PdfParser.parse() but before the text reaches book_to_skill/sanitize.py. This ensures you work with raw extracted text before normalization removes potentially useful whitespace patterns.

Will future versions of book-to-skill automatically preserve Markdown formatting?

Based on the current architecture in book_to_skill/cli.py and the reliance on external pdftotext calls, automatic preservation of Markdown structures would require either switching to a layout-aware PDF parser (such as pdfplumber or PyMuPDF) or adding a dedicated markup reconstruction layer. As of the current implementation, no such feature exists in the repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}