Detecting Structural Headings as Chapters in Markdown, AsciiDoc, and RST: A 10-Step Heuristic Approach

The book-to-skill library identifies structural chapters by applying a 10-step heuristic in _structural_chapter_count that filters out code fences, validates heading syntax, and selects the shallowest heading depth containing at least two distinct titles.

Detecting structural headings as chapters in plain-text markup formats requires distinguishing real document structure from noise like table borders, list items, and comments inside code blocks. The book-to-skill repository implements a robust detection algorithm in book_to_skill/utils.py that processes Markdown, AsciiDoc, and reStructuredText files to identify genuine chapter boundaries. This approach combines line-by-line parsing with depth-based heuristics to accurately count chapters even in documents with mixed formatting styles.

The Core Detection Algorithm

The entry point for chapter detection is the _structural_chapter_count function spanning lines 995–1072 in book_to_skill/utils.py. This function implements a multi-pass analysis that treats headings as structural chapters only when they satisfy strict validation criteria.

Step-by-Step Validation Criteria

The algorithm processes text through the following sequential filters:

• Line Isolation: The input text splits into individual lines using text.splitlines() at line 995, enabling line-number-based exclusion logic.

• Code Fence Exclusion: Before analyzing headings, the algorithm calls _closed_fence_line_numbers (lines 332–357) to identify and skip any lines contained within properly closed fenced code blocks (delimited by triple backticks or tildes).

• Setext and RST Underline Detection: For reStructuredText and Markdown Setext styles, the algorithm recognizes level-1 headings (= underline) and level-2 headings (- underline) at lines 1028–1042. A valid underline must match the title length, consist solely of the underline character, and follow a non-blank title containing word characters.

• ATX and AsciiDoc Prefix Detection: Lines 1045–1055 handle hash-prefixed Markdown (# Title) and equal-prefixed AsciiDoc (= Title) headings using the _ATX_HEADING regex pattern defined at lines 317–326.

• Content Validation: Titles must contain at least one word character (\w). Empty titles or punctuation-only strings (like =====) are discarded at line 1051 via the condition if title and re.search(r"\w", title).

• Numeric Title Separation: Headings beginning with digits are diverted into a temporary numbered map at line 1054 for specialized structural analysis, preventing list items like "1. Introduction" from being misclassified.

• Distinct Title Aggregation: Valid titles aggregate into a levels dictionary at line 1056, storing unique lower-cased titles per depth to detect genuine document structure versus repetitive headers.

• Numeric Title Heuristic: The _numbered_titles_are_structural function (lines 73–92) evaluates digit-led headings at lines 1061–1063, requiring at least three numbered headings with a median body length ≥ 200 characters to qualify as chapters.

• Depth Selection: The final chapter count derives from the shallowest heading depth containing at least two distinct titles (loop at lines 1065–1068), correctly identifying the chapter level in typical # Book Title / ## Chapter hierarchies.

• Thin Document Fallback: If no depth satisfies the two-title minimum, the function returns the total distinct title count across all depths (lines 1069–1071).

Handling Code Fences and Noise

The algorithm explicitly excludes content inside fenced code blocks to prevent false positives from commented code. The _closed_fence_line_numbers helper tracks opening and closing fence markers, returning a set of line indices that _structural_chapter_count skips during heading detection. This ensures that a Python comment like # This is not a chapter inside a triple-backtick block is ignored, while a real ATX heading outside the fence is processed.

Format-Specific Heading Patterns

The detection logic accommodates three distinct markup conventions through regex patterns defined at lines 317–326:

• Markdown ATX: Lines beginning with 1–6 hash symbols followed by a space and title text.
• AsciiDoc: Lines beginning with 1–6 equal signs followed by a space and title text.
• RST Setext: Title lines followed by underline strings of matching length consisting solely of = or - characters.

All formats require the title to contain word characters, eliminating table borders and decorative separators that consist only of punctuation.

The Numbered Title Heuristic

Digit-only headings receive special treatment to distinguish list numbering from chapter numbering. The _numbered_titles_are_structural function applies two strict requirements: a minimum of three numbered headings must exist, and the median content length between headings must exceed 200 characters. This prevents short lists ("1. Item", "2. Item") from inflating chapter counts while preserving legitimate numeric chapters ("Chapter 1", "Chapter 2") in lengthy documents.

How to Use detect_structure in Your Code

The detect_structure function exposes the chapter detection logic for practical use. Below are examples demonstrating detection across all three supported formats.

from book_to_skill.utils import detect_structure

# Markdown example with mixed content

md = """

# My Book Title

## Chapter 1

Content of the first chapter with sufficient length to establish structure.

## Chapter 2

More content here.

```python

# This is inside a fenced code block and should be ignored

def foo():
    pass

3. Not a chapter

Just a list item. """

info = detect_structure(md) print(info["chapters_detected"]) # → 2

print(info["chapters_method"]) # → "structural"

```python

# AsciiDoc example

adoc = """
= Book Title
:toc:

== Chapter One
Introduction text that provides context for the first chapter.

== Chapter Two
Additional content expanding on the second topic.
"""

info = detect_structure(adoc)
print(info["chapters_detected"])   # → 2

# RST (setext) example

rst = """
Book Title
==========

Chapter 1
---------

Content for the first section with detailed explanations.

Chapter 2
---------

More content for the second section.
"""

info = detect_structure(rst)
print(info["chapters_detected"])   # → 2

Summary

• The _structural_chapter_count function in book_to_skill/utils.py implements a 10-step heuristic for detecting structural headings as chapters across Markdown, AsciiDoc, and RST formats.
• Code fence detection via _closed_fence_line_numbers prevents false positives from commented code inside triple-backtick blocks.
• Valid headings must contain word characters (\w) and appear at the correct depth with at least one sibling title to qualify as structural chapters.
• Numbered titles undergo additional validation requiring three instances and substantial body content (≥ 200 characters median) to avoid misclassifying list items.
• The algorithm selects the shallowest heading depth containing ≥ 2 distinct titles, ensuring # Book Title / ## Chapter hierarchies resolve correctly.

Frequently Asked Questions

How does book-to-skill distinguish between table borders and real headings?

The algorithm rejects titles that lack word characters. Table borders like =====() or ----- fail the re.search(r"\w", title) check at line 1051, while genuine headings containing alphanumeric content pass validation. Additionally, Setext underline detection requires the preceding line to contain word characters and match the underline length, eliminating most decorative borders.

Why does the algorithm ignore headings inside fenced code blocks?

The _closed_fence_line_numbers function (lines 332–357) pre-identifies all line indices contained within fenced code blocks (delimited by triple backticks or tildes). The main detection loop skips these indices, ensuring that ATX-style comments inside Python docstrings or shell scripts are not misinterpreted as document chapters.

What makes a numbered title qualify as a structural chapter?

Digit-led headings must satisfy the _numbered_titles_are_structural heuristic (lines 73–92). This requires at least three numbered headings to exist in the document, with a median body length of at least 200 characters between them. This threshold distinguishes legitimate chapter numbering ("1. Introduction", "2. Methods") from simple ordered lists that typically have shorter entries.

How does the algorithm choose between different heading depths?

The function implements a depth-selection logic at lines 1065–1068 that selects the shallowest heading level containing at least two distinct titles. This approach correctly identifies the chapter level in standard book structures where the H1 represents the book title and H2 elements represent individual chapters, while ignoring deeper heading levels used for subsections.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}