How book-to-skill Detects Chapters in Different Languages: A Technical Deep Dive

book-to-skill detects chapters in different languages using a cascade of language-specific regex patterns in the _match_chapter_number function, supporting everything from English "Chapter 1" to Chinese "第一章" and Hindi "अध्याय १", with a fallback to structural Markdown heading detection when numeric patterns fail.

The open-source repository virgiliojr94/book-to-skill extracts structured content from digital books regardless of the source language. Its chapter detection system relies on book_to_skill/utils.py, which contains specialized regular expressions for over ten major writing systems. This multilingual approach ensures the tool can segment books into logical sections even when processing mixed-script documents or non-Latin alphabets.

The Two-Pass Detection Strategy

The chapter detection algorithm operates in two sequential passes when analyzing text. This approach maximizes accuracy across different formatting conventions while maintaining processing speed.

Pure-Text Pattern Matching

During the first pass, the _chapter_number function scans each line for chapter headings that begin the line or follow standard Markdown/AsciiDoc prefixes. The system checks against a prioritized list of language-specific regex patterns implemented in _match_chapter_number. If a line matches any pattern, the function extracts the numeric identifier—either as Arabic numerals, Roman numerals, or language-specific number words—and returns it as an integer.

Markdown Prefix Fallback

If the initial pure-text scan fails, the algorithm strips common Markdown heading markers such as ## or AsciiDoc style == prefixes and retries the match. This fallback mechanism, located in _chapter_number (lines 378-405 of utils.py), ensures that formatted documents do not bypass detection simply because of syntactic markup.

Language-Specific Regex Patterns

The heart of the detection system lives in the regex definitions spanning lines 103-258 of book_to_skill/utils.py. Each pattern targets specific linguistic conventions for marking chapter divisions.

Latin Script Languages

For English, Spanish, Portuguese, French, German, Italian, Dutch, and Vietnamese, the system uses a comprehensive pattern matching terms like "chapter", "chapitre", "capítulo", "hoofdstuk", and "chương". The regex at lines 133-176 captures both Arabic numerals (\d{1,2}) and Roman numerals ([IVXLCDMivxlcdm]{1,7}), allowing detection of headings like "Chapter 5: Introduction", "Capítulo II. El principio", or "Ch. 10".

CJK Scripts

Chinese support handles both simplified and traditional characters through patterns recognizing "第 N 章/回/卷/节/篇/讲" structures. The regex at lines 185-200 captures Chinese numerals including "零一二两三四五六七八九十百千" and full-width digits. A separate Markdown-aware pattern (lines 201-214) handles prefixed headings like ## 第 一讲 · 概论.

Korean detection (lines 250-258) identifies "제 N 장/편/절/관" formats, matching headings such as "제 4장 근로시간과 휴식" while accounting for optional ordinal suffixes.

Indic Scripts

Hindi and Bengali implementations use Devanagari and Bengali script recognition respectively. The Hindi pattern at lines 217-225 matches "अध्याय N" (e.g., "अध्याय १"), while the Bengali equivalent at lines 227-236 detects "অধ্যায় N" (e.g., "অধ্যায় ৩"). Both patterns include optional Markdown prefix handling and language-specific digit ranges ([०-९] for Hindi, [০-৯] for Bengali).

Thai and Persian Support

Thai detection (lines 239-247) recognizes "บทที่ N", "ตอนที่ N", and "ภาคที่ N" formats using Thai digits ([0-9๐-๙]+). Persian support implements complex logic in the separate _fa_chapter_number function to handle Arabic-Indic digits, Persian ordinals, and glued PDF text artifacts commonly found in Persian publications.

The Core Detection Algorithm

The detect_structure function (lines 608-639 of utils.py) orchestrates the entire process. It iterates through document lines, collecting distinct chapter numbers and sample headings:

def detect_structure(text: str) -> dict:
    numbers = set()
    headings = []
    
    for line in text.splitlines():
        num = _chapter_number(line)          # tries all language patterns

        if num is not None:
            numbers.add(num)                 # distinct chapter numbers only

            headings.append(line.strip())
    
    # Determine detection method

    if len(numbers) >= 2:
        method = "numeric"
    else:
        method = "structural"
        chapters_detected = _structural_chapter_count(text)
    
    return {
        'chapters_detected': len(numbers) if method == "numeric" else chapters_detected,
        'chapters_method': method,
        'chapter_headings_sample': headings[:5],
        'has_toc': False
    }

When fewer than two distinct numeric headings are found, the system falls back to _structural_chapter_count, which counts generic Markdown/AsciiDoc headings without requiring specific chapter keywords.

Implementation Reference Points

All detection logic resides in book_to_skill/utils.py with the following key locations:

• Chapter-type regexes: Lines 103-112, 133-176, 185-200, 201-214, 217-225, 227-236, 239-247, and 250-258
• _match_chapter_number: Lines 221-275
• _chapter_number (Markdown prefix handling): Lines 378-405
• detect_structure (overall orchestration): Lines 608-639

The parsing pipeline begins in book_to_skill/parsers/text.py, which provides the read_text_file helper used before structure detection. Unit tests in tests/test_chapter_method_reported.py verify multilingual detection accuracy across supported scripts.

Practical Usage Examples

The detect_structure function accepts raw text strings and returns detection metadata:

from book_to_skill.utils import detect_structure

# English example

english = """\
Chapter 1: Introduction
...
Chapter 2: Methods
"""
print(detect_structure(english)['chapters_detected'])

# → 2 (numeric)

# Spanish example

spanish = """\
Capítulo 1 – Visión general
...
Capítulo 2 – Detalles técnicos
"""
print(detect_structure(spanish)['chapters_detected'])

# → 2 (numeric)

# Chinese example

chinese = """\
第 一章 起航
...
第 二章 航行
"""
print(detect_structure(chinese)['chapters_detected'])

# → 2 (numeric)

# Thai example

thai = """\
บทที่ 1 การเริ่มต้น
...
บทที่ 2 การทำงานต่อ
"""
print(detect_structure(thai)['chapters_detected'])

# → 2 (numeric)

# Korean example

korean = """\
제 1장 서론
...
제 2장 본문
"""
print(detect_structure(korean)['chapters_detected'])

# → 2 (numeric)

Each call returns a dictionary containing chapters_detected (integer count), chapters_method (either "numeric" or "structural"), chapter_headings_sample (list of up to 5 heading strings), and has_toc (boolean).

Summary

• book-to-skill detects chapters in different languages using regex patterns tailored to specific scripts and linguistic conventions in book_to_skill/utils.py.
• The two-pass algorithm first attempts pure-text matching, then strips Markdown prefixes and retries if initial detection fails.
• Supported scripts include Latin (English, Spanish, French, etc.), CJK (Chinese, Korean), Indic (Hindi, Bengali), Thai, and Persian, each with dedicated regex patterns handling both Arabic and native numerals.
• The detect_structure function returns detection metadata including method type ("numeric" vs "structural") and sample headings.
• A structural fallback counts generic Markdown headings when fewer than two numeric chapter headings are identified.

Frequently Asked Questions

How does book-to-skill handle books with mixed languages?

The detection system processes each line independently against all language patterns simultaneously. Since patterns for different scripts use mutually exclusive Unicode ranges, a document containing both English "Chapter 1" and Chinese "第二章" will detect chapters from both languages, though the chapters_method classification depends on the dominant pattern found.

What happens if a book uses Roman numerals instead of Arabic numbers?

The Latin script regex explicitly captures Roman numerals via the (?P<roman>[IVXLCDMivxlcdm]{1,7}) named group. Headings like "Chapter IV" or "Part II. The Beginning" match successfully and convert to integers (4 and 2 respectively) for the numeric detection method.

Can the system detect chapters in PDF files with corrupted text encoding?

The Persian-specific _fa_chapter_number function includes logic to handle "glued PDF text" artifacts where spaces between words have been lost during extraction. For other languages, the regex patterns use word boundaries (\b) and flexible whitespace matching (\s*) to accommodate minor encoding irregularities.

Where is the chapter detection logic located in the repository?

All multilingual chapter detection code resides in book_to_skill/utils.py between lines 103-405, with the primary orchestration function detect_structure located at lines 608-639. The regex patterns span lines 103-258, while helper functions like _chapter_number and _match_chapter_number occupy lines 221-275 and 378-405 respectively.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}