How book-to-skill Handles Non-Latin Scripts During Chapter Detection

The book-to-skill library detects chapters by parsing Markdown heading syntax using Unicode-aware regex patterns that match any non-whitespace character, enabling native support for CJK, Cyrillic, Arabic, and other non-Latin scripts without requiring language-specific configuration.

Processing educational content in global languages requires robust text parsing that does not assume ASCII characters. The book-to-skill open-source project addresses this by implementing chapter detection logic that relies on structural Markdown markers rather than English keywords. This approach ensures that documents written in non-Latin scripts—such as Chinese, Japanese, or Arabic—are processed correctly as long as they follow standard Markdown heading conventions.

Structural Chapter Detection via Markdown Syntax

The primary chapter detection mechanism in book-to-skill ignores the linguistic content of headings and instead identifies chapter boundaries by recognizing Markdown formatting patterns. This method is inherently script-agnostic because it validates punctuation and whitespace rather than alphabetic characters.

ATX Headings (Hash-Based Markers)

The parser identifies ATX-style headings—lines beginning with one to six hash characters (#) followed by a space. In book_to_skill/utils.py, the _structural_chapter_count function processes these lines using a regex that treats the text following the hash as the heading content. Because this pattern utilizes the \S character class to match non-whitespace tokens, it accepts any Unicode character, including those from CJK supplementary planes or right-to-left scripts.

Setext Headings (Underlined Text)

Setext-style headings are detected when a line of text is immediately followed by a line consisting solely of equal signs (=) or hyphens (-). This structural detection method is completely agnostic to the script used in the heading text, whether Latin, Cyrillic, or Devanagari.

The Core Implementation in book_to_skill/utils.py

The _structural_chapter_count function serves as the central mechanism for counting chapters during text extraction. According to the source code analysis, this implementation scans extracted plain text to identify the two Markdown heading types described above. By relying on pattern matching for structural markers rather than linguistic keywords, the utility avoids the limitations of ASCII-centric parsing and supports the full Unicode range in heading content.

Optional Keyword-Based Fallback in tools/evals/paper_flat.py

An alternative regex-based approach exists in tools/evals/paper_flat.py for evaluation purposes. This pattern specifically matches lines containing the English words "chapter" or "chap." followed by Arabic numerals or Roman numerals ([ivxlcdm]+):

r"^(?:#{1,6}\s+.+|(?:chapter|chap\.)\s+(?:\d+|[ivxlcdm]+)\b.*)$"

Because this auxiliary detection method requires the literal ASCII strings "chapter" or "chap.", it does not support non-Latin terminology such as "章节" (Chinese) or "глава" (Russian). Primary chapter extraction should rely on the Markdown-based detection in utils.py for internationalization support.

Testing Non-Latin Script Support

The test suite includes tests/test_cjk_supplementary_plane.py, which specifically validates chapter detection in documents containing Chinese characters. This test exercises the _structural_chapter_count logic against content from the CJK supplementary plane, verifying that the regex correctly counts chapters when headings contain non-Latin scripts.

Practical Example: Processing CJK Documents

The following example demonstrates how book-to-skill handles a Markdown file with Chinese chapter headings without requiring additional configuration:

# Example markdown with Chinese chapter headings

cat > demo.md <<'EOF'

# 第1章 引言

内容…

# 第2章 方法

内容…

# 第3章 结果

内容…
EOF

# Run the extractor; it will split the file into three chapters

python -m book_to_skill.cli extract demo.md --output-dir out

Running the command above produces three files under out/chapters/. The headings contain Chinese characters, yet the chapter splitter works without any extra configuration because it parses the # markers.

Key Files Reference

File	Role
book_to_skill/utils.py	Implements _structural_chapter_count, the core chapter-counting logic that parses ATX and Setext headings.
book_to_skill/parsers/text.py	Handles plain-text (Markdown) parsing; invokes the utils routine for chapter detection.
tools/evals/paper_flat.py	Provides an auxiliary regex for “chapter” keyword splitting (used only by the evaluation helper).
tests/test_cjk_supplementary_plane.py	Unit test verifying correct chapter counting when headings are written in Chinese.

Summary

• Markdown-based detection: The library identifies chapters via ATX (#) and Setext (=/-) heading syntax, not language-specific keywords.
• Unicode regex: The _structural_chapter_count function in book_to_skill/utils.py uses \S patterns that support all non-whitespace Unicode characters.
• No configuration required: Non-Latin scripts including CJK, Cyrillic, and Arabic are supported natively without special flags or language packs.
• Fallback limitations: The optional regex in tools/evals/paper_flat.py only matches English "chapter" keywords and should not be relied upon for international documents.
• Tested compliance: The tests/test_cjk_supplementary_plane.py test verifies correct handling of Chinese characters in production code.

Frequently Asked Questions

Does book-to-skill require language packs to handle Arabic or Hebrew text?

No. Because the _structural_chapter_count function in book_to_skill/utils.py detects structural Markdown markers rather than parsing natural language, it processes Arabic, Hebrew, and other right-to-left scripts automatically. The regex uses Unicode-aware \S classes that treat all non-whitespace characters equally, regardless of script directionality.

What happens if my document uses the word "章节" instead of Markdown headings?

If the document lacks Markdown heading syntax—such as lines starting with # or underlined with =—the library will not detect chapters via the primary method. The auxiliary regex in tools/evals/paper_flat.py specifically looks for ASCII strings "chapter" or "chap.", so documents relying on Chinese terms like "章节" without Markdown formatting will not be automatically segmented into chapters.

Is the chapter detection regex case-sensitive for the optional keyword matching?

The optional detection in tools/evals/paper_flat.py uses a case-insensitive match for "chapter" and "chap." when that specific evaluation helper is invoked. However, the primary Markdown detection in _structural_chapter_count has no case sensitivity concerns because it matches structural characters (#) that have no case.

How does the library handle combining characters in Indic scripts within headings?

The regex patterns in book_to_skill/utils.py match grapheme clusters as sequences of non-whitespace characters, meaning combining characters used in Devanagari or Tamil are preserved within the heading text. As long as the structural Markdown markers are present, the specific Unicode composition of the heading content does not affect detection accuracy.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}