How Book-to-Skill Analyzes the Structure of a Document: Multi-Language Chapter Detection

Book-to-Skill analyzes document structure using a two-phase approach in detect_structure(): it first searches for language-specific numeric chapter headings across more than a dozen languages, then falls back to structural heading-depth analysis if fewer than two chapters are detected, while simultaneously scanning the first 30,000 characters for a table of contents.

Book-to-Skill is an open-source Python tool that converts books into structured skill trees by intelligently parsing document hierarchies. To accurately analyze the structure of a document regardless of format—from PDFs to EPUBs—the library employs a sophisticated, multilingual parsing engine centered in book_to_skill/utils.py. This system processes raw text through regex-based pattern matching for diverse writing systems alongside heuristic structural analysis.

The Core Detection Algorithm: detect_structure()

The detect_structure() function serves as the single entry point for structural analysis, processing raw text extracted from any source format. It operates in two distinct phases: numeric chapter detection and structural heading analysis. The function returns a dictionary containing chapters_detected, chapters_method (either "numeric" or "structural"), and has_toc.

Phase 1: Numeric Chapter Detection

The primary method for analyzing document structure relies on _chapter_number() and _match_chapter_number() to identify explicitly numbered chapters. This system supports multiple language families through dedicated regex patterns stored in book_to_skill/utils.py.

European Language Support

For English, French, German, Italian, Dutch, and Vietnamese, the _EXPLICIT_CHAPTER regex matches keywords like chapter, capítulo, kapitel, and chương followed by 1- or 2-digit Arabic numerals or Roman numerals. The pattern handles various punctuation styles, including Chapter 5 and Capítulo 3:.

CJK (Chinese, Japanese, Korean) Handling

East Asian documents are parsed using dedicated patterns: _CN_CHAPTER and _MD_CN_HEADING recognize constructs like "第 N 章" (Chapter N) and "第 N 节" (Section N), while _KO_CHAPTER handles Korean formats such as "제 N 장". These regexes also accommodate full-width Arabic digits (０‑９) and Markdown heading prefixes.

South Asian and Persian Scripts

For Thai, Hindi, and Bengali, the system uses _TH_CHAPTER, _HI_CHAPTER, and _BN_CHAPTER to map native digits to ASCII equivalents, detecting headings prefixed with words like บทที่ (Thai), अध्याय (Hindi), and অধ্যায় (Bengali). Persian (Farsi) support via _FA_CHAPTER is particularly robust, utilizing a _fa_ordinal_map dictionary to parse both digit forms and word ordinals up to 34 (e.g., "فصل اول" for "First Chapter").

Roman Numerals and Markdown

The _ROMAN_HEAD and _LC_MD_ROMAN patterns capture uppercase Roman numerals within ATX-style headings (## I. Intro), while restricting lowercase Roman numerals to Markdown contexts only to avoid false positives in prose.

Guarding Against False Positives

To prevent misidentifying cross-references like "Chapter 6 explores…" as headings, the _HEADING_TAIL regex verifies that matched lines end with punctuation, contain title case words, or terminate appropriately. Additionally, all discovered chapter numbers are stored in a Python set() to deduplicate entries that might appear in both the table of contents and the actual chapter headings.

If two or more distinct numeric headings are found, detect_structure() returns chapters_method = "numeric" with the corresponding count.

Phase 2: Structural Heading Analysis

When numeric detection yields fewer than two chapters (numeric_count < 2), the system invokes _structural_chapter_count() as a fallback. This method analyzes document hierarchy through markup patterns:

• ATX headings (# Title, ## Section) are matched using _ATX_HEADING

• Setext/RST underline headings (Title\n===) are identified by _SETEXT_UNDERLINE

• Headings inside fenced code blocks are excluded using _closed_fence_line_numbers

The algorithm groups titles by depth (the number of leading # or = characters) and returns the count of distinct titles at the shallowest depth containing at least two unique headings—typically the chapter level in a properly structured document. If no depth meets this criterion, the function sums all distinct titles across depths.

Table of Contents Detection

Concurrent with chapter analysis, the system scans the first approximately 30,000 characters for a table of contents using the multilingual _TOC_PATTERN. This regex includes English/European terms ("table of contents", "indice", "inhaltsverzeichnis") and CJK patterns ("目 录"), setting has_toc = True when detected.

Integration Across Document Formats

The detection logic is format-agnostic. All parsers—whether handling HTML (extract_html_content() in book_to_skill/parsers/html.py), PDF, EPUB, DOCX, or RTF—extract plain text and pass it to detect_structure(). This unified approach ensures consistent chapter counting and ToC identification regardless of source format.

Practical Implementation Examples

Direct Python API Usage

from book_to_skill.utils import detect_structure

text = """

# My Book

## Chapter 1: Introduction

Lorem ipsum…

## Chapter 2: Methods

Dolor sit…

Table of contents
1. Chapter 1
2. Chapter 2
"""

result = detect_structure(text)
print(result)

# {'chapters_detected': 2,

#  'chapters_method': 'numeric',

#  'chapter_headings_sample': ['## Chapter 1: Introduction',

#                               '## Chapter 2: Methods'],

#  'has_toc': True}

Command-Line Interface

python -m book_to_skill --mode pdf path/to/book.pdf

The CLI (book_to_skill/cli.py) extracts text and invokes the detector, exposing chapter counts and ToC flags to the skill-generation pipeline.

Processing CJK Documents

from book_to_skill.parsers.epub import extract_epub_file

plain = extract_epub_file('中文書.epub')
info = detect_structure(plain)
print(info['chapters_detected'], info['chapters_method'])

# Output might show structural detection for Chinese headings

Summary

• Primary Method: Book-to-Skill analyzes document structure through detect_structure() in book_to_skill/utils.py, which prioritizes numeric chapter detection across more than a dozen languages.
• Language Coverage: The system includes dedicated regexes for European languages, CJK scripts (Chinese, Japanese, Korean), South Asian scripts (Thai, Hindi, Bengali), and Persian (Farsi), including Roman numerals and full-width digits.
• Fallback Strategy: When fewer than two numeric chapters are found, the system falls back to structural analysis of heading depths (ATX and Setext styles).
• ToC Detection: A multilingual pattern scans the first 30,000 characters to identify tables of contents.
• Format Agnostic: The same detection engine processes HTML, PDF, EPUB, DOCX, and plain text through a unified interface.

Frequently Asked Questions

How does Book-to-Skill handle documents without explicit chapter numbers?

When fewer than two numeric headings are detected, the system automatically falls back to _structural_chapter_count(), which analyzes heading hierarchy by depth. It identifies the shallowest heading level containing at least two unique titles (typically ## in Markdown or === underlines), treating this as the chapter level for structural analysis.

Which languages are supported for chapter detection?

The library supports explicit chapter detection for English, French, German, Italian, Dutch, Vietnamese, Chinese, Japanese, Korean, Thai, Hindi, Bengali, and Persian (Farsi). Each language uses dedicated regex patterns in book_to_skill/utils.py to match culturally specific heading formats, including full-width digits for CJK scripts and native digit words for Persian ordinals.

Can Book-to-Skill distinguish between actual chapter headings and references to chapters in the text?

Yes. The _HEADING_TAIL regex acts as a prose guard, verifying that matched lines end with punctuation, contain capitalized title words, or terminate appropriately. This prevents sentences like "Chapter 6 explores quantum mechanics" from being misidentified as chapter breaks, while legitimate headings like "## Chapter 6: Quantum Mechanics" are accepted.

Where is the chapter detection logic located in the source code?

All core structural analysis logic resides in book_to_skill/utils.py, specifically within the detect_structure() function and its helpers: _chapter_number(), _match_chapter_number(), and _structural_chapter_count(). The CLI entry point in book_to_skill/cli.py and various format-specific parsers in book_to_skill/parsers/ (html.py, pdf.py, epub.py, docx.py) feed extracted text into this central utility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}