Chunking Strategy for Content Processing in Open Notebook: A Deep Dive into utils/chunking.py

Open Notebook's utils/chunking.py implements a four-stage pipeline that converts raw text into token-bounded chunks using environment-driven configuration, content-type detection, language-specific LangChain splitters, and secondary filtering to ensure optimal embedding quality.

The lfnovo/open-notebook repository provides a robust text processing pipeline designed to prepare diverse content formats for vector embeddings. At the heart of this system lies open_notebook/utils/chunking.py, which implements a sophisticated chunking strategy for content processing that balances semantic coherence with strict token limits. This article examines the implementation details, configuration options, and splitting algorithms that power the ingestion workflow.

Environment-Based Configuration for Token Limits

The chunking system reads three environment variables to establish processing boundaries. These settings ensure predictable token counts crucial for downstream embedding models while allowing deployment-specific tuning.

Chunk Size and Overlap Parameters

The module retrieves core parameters through dedicated helper functions:

• _get_chunk_size() (lines 33-58): Returns the maximum tokens per chunk, defaulting to 400 tokens
• _get_chunk_overlap() (lines 60-86): Returns the overlap percentage between chunks, defaulting to 15%
• _get_min_chunk_size() (lines 88-112): Returns the minimum viable chunk size, defaulting to 5 tokens

These functions enforce sane limits and parse integer values from the environment, falling back to safe defaults when variables are unset.

Hybrid Content Type Detection

Before splitting, the system identifies whether content is HTML, Markdown, or plain text using a two-phase detection strategy implemented in detect_content_type() (lines 222-263).

Extension-Based Detection

The detect_content_type_from_extension() function (lines 73-91) maps file extensions to ContentType enumerations. This provides immediate classification for files with standard extensions like .html, .md, or .txt.

Heuristic Analysis

When extensions are missing or generic, detect_content_type_from_heuristics() (lines 95-128) analyzes content samples. The system calculates confidence scores using:

• _calculate_html_score() (lines 130-178): Detects HTML tags and structure
• _calculate_markdown_score(): Identifies Markdown syntax patterns

If heuristic confidence exceeds 0.8, the system overrides generic extensions, ensuring proper handling of mislabeled files or raw web content.

Language-Specific Splitting Algorithms

Based on the detected ContentType, the pipeline instantiates specialized LangChain splitters to preserve document structure while respecting token limits.

HTML Header-Based Splitting

For HTML content, the system uses HTMLHeaderTextSplitter (lines 665-672), configured to split on <h1>, <h2>, and <h3> tags. This approach maintains semantic boundaries defined by the document's header hierarchy, keeping related content together in single chunks where possible.

Markdown Structure Preservation

Markdown files trigger MarkdownHeaderTextSplitter (lines 675-682), which respects #, ##, and ### headers. This strategy ensures that logical sections remain intact during the embedding preparation process, preserving the document's organizational intent.

Recursive Character Splitting for Plain Text

For plain text or unclassified content, the system falls back to RecursiveCharacterTextSplitter (lines 688-695). This splitter uses the configured CHUNK_SIZE, CHUNK_OVERLAP, and a token-count length function from open_notebook/utils/token_utils.py to create uniform chunks without structural dependencies.

Secondary Chunking and Quality Filtering

The pipeline includes safety mechanisms to handle edge cases where primary splitters produce oversized or trivial chunks.

Oversized Chunk Handling

The _apply_secondary_chunking() function (lines 398-415) processes HTML and Markdown outputs. When semantic splitters create chunks exceeding CHUNK_SIZE, this function re-splits the oversized content using the plain-text recursive splitter, ensuring no chunk violates token limits.

Minimum Size Filtering

Before returning results, the system filters chunks in the final output stage (lines 782-892). Chunks smaller than MIN_CHUNK_SIZE (5 tokens by default) are dropped unless they represent the sole content chunk. This prevents useless embeddings from whitespace or fragmentary text while preserving critical short content.

Practical Implementation Example

The chunk_text() function (lines 418-495) orchestrates the entire pipeline. Here are practical usage patterns:

from open_notebook.utils.chunking import chunk_text, ContentType

# Example 1 – plain text with automatic detection

plain = "Lorem ipsum " * 200   # roughly 300 tokens

plain_chunks = chunk_text(plain)
print(f"Plain chunks: {len(plain_chunks)}")

# Example 2 – Markdown file with extension detection

md_path = "docs/overview.md"
with open(md_path) as f:
    md_text = f.read()
md_chunks = chunk_text(md_text, file_path=md_path)
print(f"Markdown chunks: {len(md_chunks)}")

# Example 3 – Force HTML splitter for raw web pages

html = "<html><body><h1>Title</h1><p>Long paragraph …</p></body></html>"
html_chunks = chunk_text(html, content_type=ContentType.HTML)
print(f"HTML chunks: {len(html_chunks)}")

Summary

• Configurable token boundaries: The system uses environment variables to set chunk size (400), overlap (15%), and minimum size (5), with strict validation in _get_chunk_size(), _get_chunk_overlap(), and _get_min_chunk_size().
• Intelligent content detection: Hybrid logic in detect_content_type() combines extension mapping and heuristic analysis (0.8 confidence threshold) to correctly classify HTML, Markdown, and plain text.
• Structure-aware splitting: Specialized LangChain splitters preserve semantic boundaries—HTMLHeaderTextSplitter for <h1>-<h3>, MarkdownHeaderTextSplitter for headers, and RecursiveCharacterTextSplitter for unstructured text.
• Robust size enforcement: Secondary chunking via _apply_secondary_chunking() and minimum-size filtering ensure all output chunks meet token limits while avoiding embedding noise.

Frequently Asked Questions

What is the default chunk size in Open Notebook?

The default chunk size is 400 tokens, with a 15% overlap between consecutive chunks and a 5-token minimum threshold. These values are hardcoded as fallbacks in open_notebook/utils/chunking.py but can be overridden via environment variables read by _get_chunk_size(), _get_chunk_overlap(), and _get_min_chunk_size().

How does the system handle files with incorrect extensions?

When the extension is generic or missing, detect_content_type_from_heuristics() analyzes content samples to calculate HTML and Markdown confidence scores. If either score exceeds 0.8, the system overrides the extension-based classification, ensuring proper splitting strategy selection even for mislabeled files or raw web content pasted as text.

What happens if a semantic chunk exceeds the token limit?

The _apply_secondary_chunking() function (lines 398-415) catches oversized chunks from HTML or Markdown splitters and reprocesses them using the recursive plain-text splitter. This ensures no chunk exceeds the configured CHUNK_SIZE while preserving as much semantic structure as possible.

Where is the core chunking logic implemented?

All chunking functionality resides in open_notebook/utils/chunking.py, with token counting utilities imported from open_notebook/utils/token_utils.py. The module depends on langchain-text-splitters for the HTMLHeaderTextSplitter, MarkdownHeaderTextSplitter, and RecursiveCharacterTextSplitter implementations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}