How Book-to-Skill Detects Chapters in Various Languages: A Deep Dive into the Multilingual Parsing Pipeline

Book-to-skill detects chapter boundaries by scanning plain text for explicit multilingual heading patterns—supporting English, French, German, Italian, Dutch, Spanish, Portuguese, Chinese, Japanese, Thai, and Korean—before falling back to structural analysis of Markdown or AsciiDoc headings.

The book-to-skill repository by virgiliojr94 provides a robust Python utility for extracting semantic structure from ebooks. Understanding how it handles chapter detection across diverse languages and scripts is essential for developers building multilingual document processing pipelines.

The Multilingual Chapter Detection Pipeline

The core detection logic resides in book_to_skill/utils.py, where a multi-stage pipeline processes raw text extracted by book_to_skill/parsers/text.py. The system prioritizes explicit linguistic patterns before resorting to structural heuristics.

Explicit Pattern Matching for Global Scripts

The library employs compiled regular expressions to identify chapter headings across eleven languages and writing systems. Each pattern targets locale-specific conventions:

• _EXPLICIT_CHAPTER (lines 79–86): Matches English-style headings such as "Chapter 5" and equivalent patterns in Romance and Germanic languages including French, German, Italian, Dutch, Spanish, and Portuguese.

• _ROMAN_HEAD and _LC_MD_ROMAN (lines 95–98): Captures Roman numeral headings like "I: Loomings" and "II. The Carpet-Bag" common in classical literature.

• _CN_CHAPTER and _MD_CN_HEADING (lines 124–129): Detects Chinese and Japanese headings such as "第 3 章" and Markdown variants like "## 一 · 缘起".

• _TH_CHAPTER (lines 130–138): Identifies Thai chapter markers including "บทที่ 3" and "บทที่ ๑" with native numerals.

• _KO_CHAPTER (lines 140–154): Matches Korean legal and academic formatting like "제 1장 총칙" and Markdown-prefixed variants "## 제 4장".

Number Extraction and Normalization

Once a pattern matches, the _match_chapter_number() function (lines 90–113) extracts the numeric identifier. This utility handles Arabic numerals, Roman numerals (converting them to integers), and locale-specific numeral systems, ensuring consistent chapter ordering regardless of the input script.

Structural Fallback for Markdown and AsciiDoc

When explicit chapter headings are absent, _structural_chapter_count() (lines 181–242) analyzes document structure. This function scans for ATX headings (# Title) and Setext underlines (===/---) while excluding fenced code blocks. It identifies the shallowest heading depth containing two or more titles and reports this count as the chapter total.

Table of Contents Detection

The pipeline includes a dedicated _TOC_PATTERN regex (lines 56–68) that searches the first 30KB of text for "Table of Contents" equivalents in multiple languages. This boolean flag helps downstream components distinguish between narrative content and navigational metadata.

The Detection API in Action

The detect_structure() function (lines 342–373) orchestrates the entire workflow. It iterates through each line, invoking _chapter_number() (a wrapper for _match_chapter_number() that handles optional Markdown prefixes) to collect distinct chapter identifiers.

If explicit numbers are found, chapters_detected returns the count of unique identifiers. If none are detected, the function automatically falls back to structural analysis. The API also returns a sample of the first ten headings and a has_toc boolean.

Practical usage examples:

from book_to_skill.utils import detect_structure

# English novel format

text_en = """Chapter 1
Intro...

Chapter 2
Methods..."""
print(detect_structure(text_en)['chapters_detected'])   # → 2

# Chinese ebook with traditional markers

text_cn = """第 1 章 序言

第 2 章 方法论"""
print(detect_structure(text_cn)['chapters_detected'])   # → 2

# Markdown without explicit chapter numbers

markdown = """# My Book

## Part I

### Chapter 1

Content...

### Chapter 2

Content..."""
print(detect_structure(markdown)['chapters_detected'])  # → 2 (structural fallback)

Summary

• Book-to-skill supports eleven languages including CJK (Chinese, Japanese, Korean) and Thai scripts through dedicated regex patterns in utils.py.
• The two-tier detection strategy first attempts explicit chapter heading extraction via _match_chapter_number(), then falls back to structural heading analysis for Markdown/AsciiDoc documents.
• Roman numerals and native numeral systems are normalized to integers for consistent processing.
• The detect_structure() API provides a unified interface returning chapter counts, heading samples, and TOC presence indicators.
• Comprehensive test coverage exists in tests/test_discovery_tax.py and tests/test_metadata_encoding.py.

Frequently Asked Questions

Which languages are supported by the explicit heading detection?

Book-to-skill explicitly supports English, French, German, Italian, Dutch, Spanish, Portuguese, Chinese, Japanese, Thai, and Korean. Each language has dedicated regex patterns in book_to_skill/utils.py that match culturally specific chapter markers—from Western "Chapter X" formats to Korean "제 X장" and Thai "บทที่ X" conventions.

How does book-to-skill handle ebooks without explicit chapter numbers?

When no explicit chapter patterns are detected, the library automatically invokes _structural_chapter_count() to analyze Markdown or AsciiDoc heading hierarchies. It identifies the shallowest heading level containing multiple titles and treats those as chapter boundaries, ensuring robust detection even in plain-text or minimally structured documents.

What happens if a document contains both explicit chapter headings and structural headings?

The detect_structure() function prioritizes explicit chapter numbers. If _match_chapter_number() identifies any numeric chapter identifiers (Arabic, Roman, or native numerals), it uses the count of distinct numbers as the final chapter count and does not process structural headings. This prevents double-counting and respects the author's explicit structural markers.

Where is the chapter detection logic implemented in the codebase?

All core detection logic resides in book_to_skill/utils.py. Key functions include detect_structure() (the public API), _match_chapter_number() (for number extraction), and _structural_chapter_count() (for fallback analysis). Text extraction preprocessing occurs in book_to_skill/parsers/text.py, while tests/test_discovery_tax.py and tests/test_metadata_encoding.py provide multilingual validation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}