How CJK and Thai Chapter Headings Are Detected in book-to-skill: Regex Patterns Explained

The book-to-skill library detects CJK and Thai chapter headings in book_to_skill/utils.py using language-specific regular expressions that match cultural chapter markers and convert native numerals to integers.

The book-to-skill open-source project automatically converts books into structured skills by parsing chapter boundaries. Detecting CJK and Thai chapter headings requires specialized logic that recognizes Unicode numerals and language-specific classifiers like "第" or "บทที่". The detection engine implements regex-based matchers in book_to_skill/utils.py to identify these headings and normalize them into integer values for downstream processing.

CJK Detection Using _CN_CHAPTER and _MD_CN_HEADING

CJK (Chinese-Japanese-Korean) chapter detection relies on two complementary patterns defined in book_to_skill/utils.py. These patterns identify the traditional "第" prefix and various chapter classifiers while handling both plain text and Markdown-formatted lines.

Pattern Structure and Unicode Ranges

The primary CJK matcher _CN_CHAPTER validates lines against the cultural convention of numbering chapters with native numerals:

# Defined in book_to_skill/utils.py

_CN_CHAPTER = re.compile(
    rf"^\s*第\s*([0-9{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[章回卷节篇讲]"
)
_MD_CN_HEADING = re.compile(
    rf"^#{{1,6}}\s+第?\s*([{_FW_DIGITS}{_CN_NUM_CLASS}]+)\s*[·、.:：章回卷节篇讲]"
)

The pattern breakdown demonstrates precise Unicode handling:

• ^\s*第\s* — Matches optional leading whitespace followed by the CJK character "第" (indicating sequence/ordinal).
• ([0-9{_FW_DIGITS}{_CN_NUM_CLASS}]+) — Captures the chapter number composed of:
  • ASCII digits (0-9)
  • Full-width digits (０-９) common in Japanese typesetting
  • Native Chinese numerals (〇零一二三四五六七八九十百千) defined in _CN_NUM_CLASS
• \s*[章回卷节篇讲] — Requires a trailing CJK classifier such as "章" (chapter), "回" (episode), or "卷" (volume).

The _MD_CN_HEADING variant additionally accommodates Markdown syntax by allowing # heading markers (^#{1,6}\s+) and optional punctuation after the numeral.

Converting Chinese Numerals to Integers

When _CN_CHAPTER or _MD_CN_HEADING matches a line, the captured numeral group passes to _cn_numeral_to_int:

cm = _CN_CHAPTER.match(s) or _MD_CN_HEADING.match(s)
if cm:
    return _cn_numeral_to_int(cm.group(1))

This function handles both Arabic digits and native Chinese numerical composition (e.g., converting "二十五" to 25), ensuring consistent integer output regardless of input script.

Thai Detection with _TH_CHAPTER and _TH_DIGIT_MAP

Thai chapter detection operates similarly but targets the Thai writing system's unique positional digits and chapter terminology.

Thai Chapter Keywords and Regex

The _TH_CHAPTER pattern in book_to_skill/utils.py recognizes specific Thai chapter markers:

# Defined in book_to_skill/utils.py

_TH_DIGIT_MAP = str.maketrans("๐๑๒๓๔๕๖๗๘๙", "0123456789")
_TH_CHAPTER = re.compile(
    rf"^\s*(?:#{{1,6}}\s+)?(?:บทที่|ตอนที่|ภาคที่|บท|ตอน|ภาค)\s*([0-9{_TH_DIGITS}]+)\b"
)

The regex structure includes:

• (?:#{{1,6}}\s+)? — An optional non-capturing group for Markdown heading prefixes (# through ######).
• (?:บทที่|ตอนที่|ภาคที่|บท|ตอน|ภาค) — Thai chapter keywords including "บทที่" (chapter), "ตอนที่" (episode), and "ภาคที่" (part).
• ([0-9{_TH_DIGITS}]+) — Captures digits that may be Arabic (0-9) or Thai positional (๐-๙).
• \b — Word boundary anchor preventing matches within prose words.

Digit Translation Logic

Unlike CJK numerals which require compositional parsing, Thai digits map directly to Arabic equivalents. The library uses _TH_DIGIT_MAP for conversion:

tm = _TH_CHAPTER.match(s)
if tm:
    return int(tm.group(1).translate(_TH_DIGIT_MAP))

The str.translate method efficiently maps Thai digits ๐๑๒๓๔๕๖๗๘๙ to 0123456789, enabling standard Python int() conversion.

The Chapter Detection Workflow

The detection process orchestrates multiple matchers through _match_chapter_number and _chapter_number, prioritizing specificity over generality.

Primary Matching in _match_chapter_number

The _match_chapter_number function executes matchers in a defined sequence:

def _match_chapter_number(line: str) -> int | None:
    s = line.strip()
    # Latin / Roman numeral cases omitted for brevity...

    
    cm = _CN_CHAPTER.match(s) or _MD_CN_HEADING.match(s)
    if cm:
        return _cn_numeral_to_int(cm.group(1))

    tm = _TH_CHAPTER.match(s)
    if tm:
        return int(tm.group(1).translate(_TH_DIGIT_MAP))
    
    # Korean, Persian, and other language handlers...

The function strips leading/trailing whitespace and tests CJK patterns before Thai patterns, ensuring specific cultural markers take precedence over generic number matching.

Secondary Pass for Markdown Prefixes

If primary matchers fail, _chapter_number initiates a second pass that sanitizes Markdown or AsciiDoc syntax:

1. Strip optional heading prefixes (##, ==, etc.)
2. Retry _match_chapter_number on the cleaned string

This two-pass architecture ensures headings like ## 第十章 (Markdown) or == บทที่ ๕ == (AsciiDoc) resolve correctly even when raw patterns expect plain text.

Practical Implementation Examples

The following examples demonstrate detection against raw text lines:

from book_to_skill.utils import _chapter_number

# CJK examples with mixed numeral types

print(_chapter_number("第３章 引言"))          # → 3 (Full-width digit)

print(_chapter_number("## 第十章 结束"))      # → 10 (Markdown + Chinese numeral)

print(_chapter_number("第25回 事件"))         # → 25 (Arabic digits)

# Thai examples with native digit translation

print(_chapter_number("บทที่ 12"))            # → 12 (Arabic digits)

print(_chapter_number("## บทที่ ๑๕"))        # → 15 (Thai digits ๑๕ translated)

print(_chapter_number("ตอนที่ ๐๓"))          # → 3 (Leading zero preservation)

These functions return None for non-heading lines and integers for valid chapter headings, enabling the detect_structure routine to build accurate document hierarchies.

Summary

• Detection location: The logic resides in book_to_skill/utils.py within _match_chapter_number and _chapter_number.
• CJK patterns: _CN_CHAPTER and _MD_CN_HEADING match the "第" prefix and classifiers ("章", "回", etc.), with _cn_numeral_to_int handling complex Chinese numerals.
• Thai patterns: _TH_CHAPTER identifies keywords like "บทที่" and uses _TH_DIGIT_MAP to translate Thai digits (๐-๙) before integer conversion.
• Markdown support: A secondary pass strips heading markers and retries detection, ensuring compatibility with formatted documents.
• Execution order: CJK and Thai matchers run after Latin/Roman checks but before generic fallback patterns, maximizing accuracy for known linguistic structures.

Frequently Asked Questions

What source file contains the CJK and Thai chapter detection logic?

The regular expressions and matching functions reside in book_to_skill/utils.py. This file defines _CN_CHAPTER, _MD_CN_HEADING, and _TH_CHAPTER alongside the _match_chapter_number orchestration function that coordinates detection across languages.

How does the library handle Markdown-formatted chapter headings?

The engine uses two strategies: first, patterns like _MD_CN_HEADING explicitly allow # prefixes; second, if initial matching fails, _chapter_number performs a second pass that strips Markdown or AsciiDoc heading markers and retries the matchers. This ensures ## 第三章 resolves identically to 第三章.

Are complex Chinese numerals like "一百二十三" supported?

Yes. The _cn_numeral_to_int function parses native Chinese numeral composition rules, converting expressions like "二十五" (25) or "一百" (100) into integers. It handles both standalone characters and multi-digit compounds containing "十", "百", and "千".

Can Thai chapter headings use Arabic digits instead of Thai digits?

Yes. The _TH_CHAPTER pattern captures both Arabic (0-9) and Thai (๐-๙) digits in the same capture group. When Thai digits are present, _TH_DIGIT_MAP translates them to Arabic equivalents before int() conversion, allowing seamless handling of mixed or pure-Arabic inputs like "บทที่ 12".

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}