Chunking Strategy for Processing Large Documents in Open Notebook

Open Notebook processes large documents by splitting them into token-bounded chunks using a multi-stage pipeline that detects content type, applies format-specific splitters, and enforces size limits through secondary chunking to optimize embeddings.

The chunking strategy for processing large documents is central to Open Notebook's document ingestion workflow. Implemented in open_notebook/utils/chunking.py, this system combines environment-driven configuration with semantic structure preservation to ensure high-quality vector embeddings while respecting model context windows.

Environment-Based Configuration

The chunking behavior is controlled through environment variables read at import time via dedicated getter functions. These settings allow operators to tune the pipeline without modifying source code.

• OPEN_NOTEBOOK_CHUNK_SIZE: Target token count per chunk (default 400), retrieved by _get_chunk_size()
• OPEN_NOTEBOOK_CHUNK_OVERLAP: Overlap between consecutive chunks as a percentage of chunk size (default 15%), calculated by _get_chunk_overlap()
• OPEN_NOTEBOOK_MIN_CHUNK_SIZE: Minimum token threshold to retain a chunk (default 5), fetched by _get_min_chunk_size()

These values are loaded once when the module initializes, ensuring consistent behavior across the application lifecycle.

Content-Type Detection Pipeline

Before splitting, the system must determine whether to treat content as HTML, Markdown, or plain text. The detect_content_type function orchestrates this through a two-tier approach.

Extension-Based Classification

The function first consults the _EXTENSION_TO_CONTENT_TYPE mapping to classify files based on their extensions. This provides immediate type resolution for standard file formats.

Heuristic Fallback Analysis

When extensions are missing, generic, or ambiguous, the system employs content heuristics. The _calculate_html_score and _calculate_markdown_score functions analyze text signatures to detect structural patterns. If either heuristic returns a confidence score of 0.8 or higher, it overrides the extension-based classification, ensuring accurate processing of mislabeled or extensionless content.

Multi-Stage Splitting Architecture

The chunk_text function implements a cascading splitter strategy that respects document semantics while enforcing hard token limits.

Primary Format-Aware Splitters

Based on the detected content type, the system selects the appropriate LangChain splitter:

• HTML: HTMLHeaderTextSplitter segments on <h1>, <h2>, and <h3> tags to preserve document hierarchy
• Markdown: MarkdownHeaderTextSplitter splits on #, ##, and ### headers to maintain section boundaries
• Plain Text: RecursiveCharacterTextSplitter uses the configured CHUNK_SIZE, CHUNK_OVERLAP, and the token_count function from open_notebook/utils/token_utils.py to measure and segment content

Secondary Size Enforcement

Header-based splitters occasionally produce oversized chunks when document sections contain large blocks of text without structural breaks. When a chunk exceeds CHUNK_SIZE, the _apply_secondary_chunking function automatically re-processes it using RecursiveCharacterTextSplitter. This secondary pass ensures no fragment exceeds the token limit while maintaining the semantic boundaries established by the primary splitter.

Quality Filtering and Validation

After splitting, the pipeline applies strict quality controls implemented in open_notebook/utils/chunking.py. Empty or whitespace-only chunks are removed immediately. The system then enforces MIN_CHUNK_SIZE filtering, dropping fragments below the token threshold to prevent null or low-information embeddings. This filtering includes a safety mechanism that retains the original content if size constraints would otherwise eliminate all chunks.

Practical Implementation Examples

To chunk a raw string with automatic content detection:

from open_notebook.utils.chunking import chunk_text

text = "..."                     # Very long document

chunks = chunk_text(text)        # Auto-detects content type

print(f"Generated {len(chunks)} chunks")

To ensure accurate content-type detection by providing file context:

chunks = chunk_text(text, file_path="report.html")

To customize chunking parameters via environment variables:

import os
os.environ["OPEN_NOTEBOOK_CHUNK_SIZE"] = "800"   # larger chunks

os.environ["OPEN_NOTEBOOK_CHUNK_OVERLAP"] = "100"

# Import after setting variables

from open_notebook.utils.chunking import chunk_text

The test suite in tests/test_chunking.py validates this behavior across content types:

def test_html_chunking():
    html = "<h1>Title</h1><p>" + "word " * 500 + "</p>"
    chunks = chunk_text(html, file_path="sample.html")
    assert len(chunks) > 1
    assert all(len(chunk) > 0 for chunk in chunks)

Summary

• Open Notebook implements chunking in open_notebook/utils/chunking.py with configurable token limits via environment variables
• Content-type detection combines extension mapping (_EXTENSION_TO_CONTENT_TYPE) with heuristic analysis (_calculate_html_score, _calculate_markdown_score)
• Primary splitters preserve semantic structure: HTML headers, Markdown headers, or recursive character splitting for plain text
• Secondary chunking (_apply_secondary_chunking) enforces hard size limits when primary splitters produce oversized segments
• Quality filters remove empty chunks and enforce MIN_CHUNK_SIZE to prevent low-quality embeddings
• Token counting relies on open_notebook/utils/token_utils.py for accurate measurement

Frequently Asked Questions

How does Open Notebook determine the content type of a document?

Open Notebook uses the detect_content_type function, which first checks file extensions against the _EXTENSION_TO_CONTENT_TYPE mapping. If the extension is missing or ambiguous, it falls back to heuristic scoring via _calculate_html_score and _calculate_markdown_score, requiring a confidence threshold of 0.8 to override extension-based classification.

What happens if a single document section exceeds the maximum chunk size?

When header-based splitters produce chunks larger than OPEN_NOTEBOOK_CHUNK_SIZE, the _apply_secondary_chunking function automatically re-chunks the oversized segment using RecursiveCharacterTextSplitter. This ensures all output chunks respect the token limit while preserving as much structure as possible.

Can I adjust chunking parameters without modifying the source code?

Yes. Set the environment variables OPEN_NOTEBOOK_CHUNK_SIZE, OPEN_NOTEBOOK_CHUNK_OVERLAP, or OPEN_NOTEBOOK_MIN_CHUNK_SIZE before importing the chunking module. The system reads these values once at import time through _get_chunk_size(), _get_chunk_overlap(), and _get_min_chunk_size().

Where does the token counting logic reside?

The token counting functionality is implemented in open_notebook/utils/token_utils.py via the token_count function. This utility is referenced by the splitters in open_notebook/utils/chunking.py to accurately measure chunk sizes against the configured limits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}