# Chunking Strategy for Processing Large Documents in Open Notebook
> Discover the chunking strategy for processing large documents in Open Notebook. Learn how this multi-stage pipeline optimizes embeddings for efficient content analysis.
- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-22
---
**Open Notebook processes large documents by splitting them into token-bounded chunks using a multi-stage pipeline that detects content type, applies format-specific splitters, and enforces size limits through secondary chunking to optimize embeddings.**
The chunking strategy for processing large documents is central to Open Notebook's document ingestion workflow. Implemented in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py), this system combines environment-driven configuration with semantic structure preservation to ensure high-quality vector embeddings while respecting model context windows.
## Environment-Based Configuration
The chunking behavior is controlled through environment variables read at import time via dedicated getter functions. These settings allow operators to tune the pipeline without modifying source code.
- **`OPEN_NOTEBOOK_CHUNK_SIZE`**: Target token count per chunk (default 400), retrieved by `_get_chunk_size()`
- **`OPEN_NOTEBOOK_CHUNK_OVERLAP`**: Overlap between consecutive chunks as a percentage of chunk size (default 15%), calculated by `_get_chunk_overlap()`
- **`OPEN_NOTEBOOK_MIN_CHUNK_SIZE`**: Minimum token threshold to retain a chunk (default 5), fetched by `_get_min_chunk_size()`
These values are loaded once when the module initializes, ensuring consistent behavior across the application lifecycle.
## Content-Type Detection Pipeline
Before splitting, the system must determine whether to treat content as HTML, Markdown, or plain text. The `detect_content_type` function orchestrates this through a two-tier approach.
### Extension-Based Classification
The function first consults the `_EXTENSION_TO_CONTENT_TYPE` mapping to classify files based on their extensions. This provides immediate type resolution for standard file formats.
### Heuristic Fallback Analysis
When extensions are missing, generic, or ambiguous, the system employs content heuristics. The `_calculate_html_score` and `_calculate_markdown_score` functions analyze text signatures to detect structural patterns. If either heuristic returns a confidence score of 0.8 or higher, it overrides the extension-based classification, ensuring accurate processing of mislabeled or extensionless content.
## Multi-Stage Splitting Architecture
The `chunk_text` function implements a cascading splitter strategy that respects document semantics while enforcing hard token limits.
### Primary Format-Aware Splitters
Based on the detected content type, the system selects the appropriate LangChain splitter:
- **HTML**: `HTMLHeaderTextSplitter` segments on ``, ``, and `` tags to preserve document hierarchy
- **Markdown**: `MarkdownHeaderTextSplitter` splits on `#`, `##`, and `###` headers to maintain section boundaries
- **Plain Text**: `RecursiveCharacterTextSplitter` uses the configured `CHUNK_SIZE`, `CHUNK_OVERLAP`, and the `token_count` function from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) to measure and segment content
### Secondary Size Enforcement
Header-based splitters occasionally produce oversized chunks when document sections contain large blocks of text without structural breaks. When a chunk exceeds `CHUNK_SIZE`, the `_apply_secondary_chunking` function automatically re-processes it using `RecursiveCharacterTextSplitter`. This secondary pass ensures no fragment exceeds the token limit while maintaining the semantic boundaries established by the primary splitter.
## Quality Filtering and Validation
After splitting, the pipeline applies strict quality controls implemented in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py). Empty or whitespace-only chunks are removed immediately. The system then enforces `MIN_CHUNK_SIZE` filtering, dropping fragments below the token threshold to prevent null or low-information embeddings. This filtering includes a safety mechanism that retains the original content if size constraints would otherwise eliminate all chunks.
## Practical Implementation Examples
To chunk a raw string with automatic content detection:
```python
from open_notebook.utils.chunking import chunk_text
text = "..." # Very long document
chunks = chunk_text(text) # Auto-detects content type
print(f"Generated {len(chunks)} chunks")
```
To ensure accurate content-type detection by providing file context:
```python
chunks = chunk_text(text, file_path="report.html")
```
To customize chunking parameters via environment variables:
```python
import os
os.environ["OPEN_NOTEBOOK_CHUNK_SIZE"] = "800" # larger chunks
os.environ["OPEN_NOTEBOOK_CHUNK_OVERLAP"] = "100"
# Import after setting variables
from open_notebook.utils.chunking import chunk_text
```
The test suite in [`tests/test_chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/tests/test_chunking.py) validates this behavior across content types:
```python
def test_html_chunking():
html = "Title
" + "word " * 500 + "
"
chunks = chunk_text(html, file_path="sample.html")
assert len(chunks) > 1
assert all(len(chunk) > 0 for chunk in chunks)
```
## Summary
- Open Notebook implements chunking in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) with configurable token limits via environment variables
- **Content-type detection** combines extension mapping (`_EXTENSION_TO_CONTENT_TYPE`) with heuristic analysis (`_calculate_html_score`, `_calculate_markdown_score`)
- **Primary splitters** preserve semantic structure: HTML headers, Markdown headers, or recursive character splitting for plain text
- **Secondary chunking** (`_apply_secondary_chunking`) enforces hard size limits when primary splitters produce oversized segments
- **Quality filters** remove empty chunks and enforce `MIN_CHUNK_SIZE` to prevent low-quality embeddings
- Token counting relies on [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) for accurate measurement
## Frequently Asked Questions
### How does Open Notebook determine the content type of a document?
Open Notebook uses the `detect_content_type` function, which first checks file extensions against the `_EXTENSION_TO_CONTENT_TYPE` mapping. If the extension is missing or ambiguous, it falls back to heuristic scoring via `_calculate_html_score` and `_calculate_markdown_score`, requiring a confidence threshold of 0.8 to override extension-based classification.
### What happens if a single document section exceeds the maximum chunk size?
When header-based splitters produce chunks larger than `OPEN_NOTEBOOK_CHUNK_SIZE`, the `_apply_secondary_chunking` function automatically re-chunks the oversized segment using `RecursiveCharacterTextSplitter`. This ensures all output chunks respect the token limit while preserving as much structure as possible.
### Can I adjust chunking parameters without modifying the source code?
Yes. Set the environment variables `OPEN_NOTEBOOK_CHUNK_SIZE`, `OPEN_NOTEBOOK_CHUNK_OVERLAP`, or `OPEN_NOTEBOOK_MIN_CHUNK_SIZE` before importing the chunking module. The system reads these values once at import time through `_get_chunk_size()`, `_get_chunk_overlap()`, and `_get_min_chunk_size()`.
### Where does the token counting logic reside?
The token counting functionality is implemented in [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) via the `token_count` function. This utility is referenced by the splitters in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) to accurately measure chunk sizes against the configured limits.