How book-to-skill Detects Chapters in Multiple Languages: Pattern Matching Across Scripts

book-to-skill determines chapter boundaries by scanning extracted plain text for language-specific numeric patterns, using dedicated regex matchers that handle Arabic numerals, Roman numerals, CJK ideographs, and script-specific chapter words across more than ten languages.

The book-to-skill repository processes documents in diverse languages to extract structured content for skill generation. To accurately detect chapters in multiple languages, the project implements a sophisticated two-stage pattern matching system in book_to_skill/utils.py that recognizes culturally distinct heading conventions—from German Kapitel to Chinese 第N章—without requiring external language detection libraries.

The Two-Stage Detection Pipeline

The detection algorithm operates through a fast filtering stage followed by aggregation logic that validates structural integrity.

Stage 1: Fast Numeric Pattern Matching with _chapter_number()

The function _chapter_number() (located at book_to_skill/utils.py lines 62‑73) serves as the primary entry point for heading analysis. It first attempts a fast "plain numbered" pattern (e.g., 1 Introduction) for efficiency. When this fails, it delegates to _match_chapter_number(), which contains the language-specific regular expressions for deeper inspection.

Stage 2: Distinct Number Aggregation in detect_structure()

The detect_structure() function (lines 91‑99) iterates over every line of text, invoking _chapter_number() to extract numeric values. It collects distinct numeric values to determine the chapter count. The count of unique numbers becomes the numeric chapter count; if no numeric headings are identified, the system falls back to a structural algorithm implemented in _structural_chapter_count().

Language-Specific Regex Patterns

The multilingual capability derives from explicit regex definitions that capture localized chapter naming conventions.

Latin Script Languages: _EXPLICIT_CHAPTER

For English, French, German, Italian, Dutch, Spanish, and Portuguese, the _EXPLICIT_CHAPTER pattern (lines 103‑111) matches explicit chapter words followed by Arabic or Roman numerals. The pattern recognizes chapter, chapitre, kapitel, capítulo, capitolo, and hoofdstuk. The _HEADING_TAIL validation ensures matches represent actual headings rather than prose mentions.

Roman Numeral Support: _ROMAN_HEAD and _LC_MD_ROMAN

The system handles classical enumeration through _ROMAN_HEAD and _LC_MD_ROMAN (lines 118‑124), matching headings such as I: Introduction or ## i. Preface. Captured numerals convert to integers via _roman_to_int().

CJK Languages: Chinese and Japanese

Two distinct patterns handle Simplified/Traditional Chinese and Japanese:

• _CN_CHAPTER (lines 132‑140): Matches <第 N 章> style headings using full-width Arabic digits and CJK ideographs.
• _MD_CN_HEADING: Recognizes markdown formats like ## 一 · Title.

Pre-processing via _KANGXI_NUMERAL_TRANS normalizes Kangxi radicals to standard ideographs before matching.

Thai Chapter Detection: _TH_CHAPTER

The _TH_CHAPTER pattern (lines 167‑174) identifies headings such as บทที่ 3 or ตอนที่ ๑. Thai digits map to Arabic equivalents before integer conversion.

Hindi (Devanagari): _HI_CHAPTER

For Hindi documents, _HI_CHAPTER (lines 77‑84) detects अध्याय 1 or अध्याय १. The system translates Devanagari digits to ASCII before numerical processing.

Korean Chapter Patterns: _KO_CHAPTER

The _KO_CHAPTER regex (lines 89‑102) captures Korean statutes and books using 제1장, 제4장, or 제2편. It optionally handles "의 N" suffixes and relies on classifiers like 장|편|절|관.

Persian (Farsi): _fa_chapter_number()

A dedicated pipeline (lines 64‑85) handles Persian documents, parsing both digit forms (فصل ۱) and word ordinals (فصل بیست و یکم). It additionally recognizes بخش ۲: … structures.

Algorithm Optimizations and Guards

Beyond pattern matching, the system implements defensive programming to avoid false positives.

Pre-processing and Normalization

Before regex evaluation, _KANGXI_NUMERAL_TRANS normalizes archaic CJK numerals. Lines exceeding 80 characters are ignored via length guards to prevent false positives in OCR-generated noise.

Markdown and AsciiDoc Handling

When lines begin with heading markers (##, ===, etc.), the _MD_HEADING_PREFIX logic strips these prefixes and retries matching. This enables recognition of formatted headings like ## Chapter 1 across all supported languages.

Structural Fallback Analysis

If no numeric headings are discovered, _structural_chapter_count() analyzes the Markdown/AsciiDoc hierarchy (ATX, Setext, and fenced-code exclusion) to infer chapter-like divisions (referenced around lines 83‑101).

Working with the Detection API

The detect_structure() function returns a standardized metadata dictionary:

from book_to_skill.utils import detect_structure

# Mixed English and Chinese content

text = """

# Introduction

Chapter 1  The Beginning

## 第 2 章 章节标题

Chapter 3  Advanced Topics
"""

result = detect_structure(text)
print(result["chapters_detected"])   # → 3

print(result["chapters_method"])     # → "numeric"

print(result["chapter_headings_sample"])

# → ['Chapter 1  The Beginning', '## 第 2 章 章节标题', 'Chapter 3  Advanced Topics']

For direct testing of individual lines, use the low-level matcher:

from book_to_skill.utils import _chapter_number

print(_chapter_number("บทที่ 3"))        # → 3  (Thai)

print(_chapter_number("अध्याय ५"))        # → 5  (Hindi)

print(_chapter_number("第 ７ 章"))       # → 7  (Chinese)

print(_chapter_number("Kapitel 12"))    # → 12 (German)

The resulting metadata attaches to each extracted source and consolidates into metadata.json with fields chapters_detected, chapters_method (values: "numeric", "structural", or "none"), chapter_headings_sample, and has_toc.

Summary

• book-to-skill detects chapters through a two-stage pipeline: fast numeric extraction via _chapter_number() and validation via detect_structure().
• The system supports 10+ languages through dedicated regex patterns in book_to_skill/utils.py, including CJK, Devanagari, Thai, and Persian scripts.
• Roman numerals and Arabic digits are normalized across all supported writing systems.
• Length guards (80-character limit) and markdown prefix stripping prevent false positives in structured documents.
• A structural fallback analyzes document hierarchy when numeric patterns fail.
• Results expose both the detection method and sample headings to downstream skill generation pipelines.

Frequently Asked Questions

Which languages does book-to-skill support for chapter detection?

The library explicitly supports English, French, German, Italian, Dutch, Spanish, Portuguese, Chinese (Simplified/Traditional), Japanese, Thai, Hindi, Korean, and Persian. Each language implements specific regex patterns in book_to_skill/utils.py to recognize localized chapter terminology such as 第, अध्याय, บทที่, and فصل.

How does the system handle Roman numerals in chapter headings?

The _ROMAN_HEAD and _LC_MD_ROMAN patterns (lines 118‑124) capture Roman numeral prefixes in both uppercase and lowercase formats. Captured values convert to integers via _roman_to_int(), allowing the system to count chapters consistently regardless of whether the source uses Arabic or Roman enumeration.

What happens when no numeric chapter patterns are found?

When detect_structure() cannot identify numeric headings, it automatically invokes _structural_chapter_count(). This fallback analyzes Markdown/AsciiDoc structural elements—specifically ATX and Setext heading hierarchies—excluding fenced code blocks to infer logical chapter boundaries. The resulting chapters_method field reports "structural" instead of "numeric".

Can the detection handle mixed-language documents?

Yes. The _chapter_number() function evaluates each line independently against all language patterns. A document containing English "Chapter 1" followed by Chinese "第 2 章" correctly registers as containing two distinct chapters, as demonstrated in the chapter_headings_sample output array.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}