Detecting Table of Contents (TOC) in book-to-skill: Multilingual Regex Logic Explained

book-to-skill determines whether a source document contains a Table of Contents by scanning the first ~30 KB of extracted plain text for multilingual TOC headings using a compiled regex pattern defined in book_to_skill/utils.py.

The open-source repository virgiliojr94/book-to-skill provides a robust pipeline for extracting structured knowledge from books and documents. Understanding the logic for detecting Table of Contents (TOC) in book-to-skill is essential for users processing multilingual documents or optimizing extraction workflows. The detection mechanism relies on a lightweight, regex-based approach that balances accuracy with performance across nine different languages.

The Multilingual TOC Header Dictionary

At the core of the detection system lies _TOC_HEADERS, a comprehensive tuple defined in book_to_skill/utils.py at lines 302-308. This constant enumerates common TOC titles across multiple Latin-script languages to maximize match probability regardless of text extraction quality.

_TOC_HEADERS = (
    "table of contents", "contents", "índice", "sumário",   # EN / ES / PT

    "sumario",                                            # PT (no accent)

    "table des matières",                                 # French

    "inhaltsverzeichnis",                                 # German

    "indice", "sommario",                                 # Italian

    "inhoudsopgave",                                      # Dutch

)

The tuple intentionally includes both accented and unaccented variants—such as "sumário" and "sumario"—to accommodate different PDF extraction engines that may strip diacritics. This defensive programming ensures robust detection across Portuguese-language documents regardless of encoding fidelity.

CJK Pattern Support and Regex Compilation

For Chinese, Japanese, and Korean (CJK) documents, the system employs a dedicated pattern rather than the Latin-based tuple. Lines 309-311 in book_to_skill/utils.py define _TOC_CJK_PATTERN, which matches the traditional and simplified characters for "contents" (目錄, 目録, 目次).

These components merge into _TOC_PATTERN (lines 310-315), a compiled regular expression that combines the escaped header strings with the CJK pattern. The regex tolerates optional ATX-style markdown heading markers (# through ######) and uses case-insensitive, multiline matching:

_TOC_PATTERN = re.compile(
    r"^\s*(?:#{1,6}\s*)?(?:" + "|".join([*(re.escape(h) for h in _TOC_HEADERS), _TOC_CJK_PATTERN]) + r")\s*$",
    re.IGNORECASE | re.MULTILINE,
)

The re.MULTILINE flag ensures the caret (^) and dollar ($) anchors match line boundaries rather than the entire string, while re.IGNORECASE handles capitalization variations in English and German titles.

Detection Pipeline Implementation

The 30 KB Text Window

The detect_structure() function implements a critical performance optimization by limiting TOC searches to the first 30,000 characters of extracted text. According to lines 651-653 in book_to_skill/utils.py, the function slices text[:30000] before applying the pattern search.

# Look for ToC indicators in the first ~30k chars (multilingual; see _TOC_PATTERN)

has_toc = bool(_TOC_PATTERN.search(text[:30000]))

This approach assumes that Tables of Contents appear within the opening pages of books, significantly reducing computational overhead on large documents while maintaining high detection accuracy. The 30 KB window typically covers the first 10-15 pages of extracted text, sufficient for most publishing formats.

Boolean Detection Logic

The function returns a simple boolean has_toc flag. If _TOC_PATTERN.search() finds a match within the initial text window, the flag is set to True; otherwise, it remains False. This binary classification feeds directly into downstream processing decisions, such as token budgeting and structural parsing.

Multi-Source Aggregation

When processing books split across multiple PDFs or source files, book-to-skill consolidates per-source results to determine global document structure. Lines 1125-1133 in book_to_skill/utils.py use Python's built-in any() function to aggregate flags across the extracted_sources list:

# has_toc is a per-source property, so it has to be combined per source

consolidated_structure["has_toc"] = any(
    src["has_toc"] for src in extracted_sources
)

This aggregation logic ensures that fragmented books are flagged as containing a TOC if any constituent file contains recognized headers, preventing false negatives when chapters are distributed across separate files.

Practical Usage Examples

To programmatically check a text string for TOC presence:

from book_to_skill.utils import detect_structure

# Simple text containing a French TOC heading

txt = "Table des matières\n1. Introduction\n2. Chapter One\n"
info = detect_structure(txt)
print(info["has_toc"])   # → True

# A document without a recognizable TOC heading

txt2 = "Chapter 1\nOnce upon a time..."
info2 = detect_structure(txt2)
print(info2["has_toc"])  # → False

For command-line verification during real book processing:

python -m book_to_skill path/to/book.pdf --mode text

# The CLI internally calls `detect_structure()` and prints:

#    ToC : yes      # if a TOC was detected

The same _TOC_PATTERN is also reused in tools/discovery_tax.py for token-budget calculations, ensuring consistency between detection and cost-estimation workflows.

Summary

• book-to-skill detects Table of Contents using a multilingual regex pattern stored in _TOC_PATTERN that supports English, Spanish, Portuguese, French, German, Italian, Dutch, and CJK languages.
• Detection is restricted to the first 30 KB of text (text[:30000]) within the detect_structure() function to optimize performance on large documents.
• The system supports both single-file and multi-source book processing, aggregating per-source boolean flags using any() logic defined at lines 1125-1133.
• Core implementation resides in book_to_skill/utils.py with comprehensive test coverage in tests/test_book_to_skill.py and multi-source validation in tests/test_multi_source_toc.py.

Frequently Asked Questions

What languages does book-to-skill support for TOC detection?

The repository supports nine language groups through the _TOC_HEADERS tuple and _TOC_CJK_PATTERN regex: English, Spanish, Portuguese, French, German, Italian, Dutch, and CJK (Chinese, Japanese, Korean). The Portuguese implementation specifically handles both accented ("sumário") and unaccented ("sumario") variants to accommodate different text extraction engines.

Why does book-to-skill limit TOC detection to the first 30 KB of text?

Limiting the search to text[:30000] optimizes processing performance for large documents while leveraging the publishing convention that Tables of Contents consistently appear within the first few pages. This 30 KB window strikes a balance between computational efficiency and detection coverage, as TOCs rarely appear deep in scholarly or commercial texts.

How does book-to-skill handle TOC detection across multiple files?

When processing books split across multiple PDFs or text files, each source generates its own has_toc boolean during extraction. The pipeline consolidates these individual flags using any() logic, marking the entire book as having a TOC if any single file contains detected headers. This prevents false negatives when chapters are distributed across separate source files.

Where is the TOC detection logic tested?

The detection mechanism includes comprehensive test coverage in tests/test_book_to_skill.py, which validates regex matching across all supported languages. Multi-source aggregation logic is specifically tested in tests/test_multi_source_toc.py, ensuring that consolidated_structure["has_toc"] correctly reflects the union of per-source detections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}