# Chunking Strategy for Content Processing in Open Notebook: A Deep Dive into utils/chunking.py

> Explore Open Notebook's chunking strategy in utils/chunking.py. Discover how this four-stage pipeline optimizes text for quality embeddings through configuration, detection, splitters, and filtering.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-21

---

**Open Notebook's [`utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/utils/chunking.py) implements a four-stage pipeline that converts raw text into token-bounded chunks using environment-driven configuration, content-type detection, language-specific LangChain splitters, and secondary filtering to ensure optimal embedding quality.**

The `lfnovo/open-notebook` repository provides a robust text processing pipeline designed to prepare diverse content formats for vector embeddings. At the heart of this system lies [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py), which implements a sophisticated **chunking strategy for content processing** that balances semantic coherence with strict token limits. This article examines the implementation details, configuration options, and splitting algorithms that power the ingestion workflow.

## Environment-Based Configuration for Token Limits

The chunking system reads three environment variables to establish processing boundaries. These settings ensure predictable token counts crucial for downstream embedding models while allowing deployment-specific tuning.

### Chunk Size and Overlap Parameters

The module retrieves core parameters through dedicated helper functions:

- **`_get_chunk_size()`** (lines 33-58): Returns the maximum tokens per chunk, defaulting to **400 tokens**
- **`_get_chunk_overlap()`** (lines 60-86): Returns the overlap percentage between chunks, defaulting to **15%**
- **`_get_min_chunk_size()`** (lines 88-112): Returns the minimum viable chunk size, defaulting to **5 tokens**

These functions enforce sane limits and parse integer values from the environment, falling back to safe defaults when variables are unset.

## Hybrid Content Type Detection

Before splitting, the system identifies whether content is HTML, Markdown, or plain text using a two-phase detection strategy implemented in `detect_content_type()` (lines 222-263).

### Extension-Based Detection

The **`detect_content_type_from_extension()`** function (lines 73-91) maps file extensions to `ContentType` enumerations. This provides immediate classification for files with standard extensions like `.html`, `.md`, or `.txt`.

### Heuristic Analysis

When extensions are missing or generic, **`detect_content_type_from_heuristics()`** (lines 95-128) analyzes content samples. The system calculates confidence scores using:

- **`_calculate_html_score()`** (lines 130-178): Detects HTML tags and structure
- **`_calculate_markdown_score()`**: Identifies Markdown syntax patterns

If heuristic confidence exceeds **0.8**, the system overrides generic extensions, ensuring proper handling of mislabeled files or raw web content.

## Language-Specific Splitting Algorithms

Based on the detected `ContentType`, the pipeline instantiates specialized LangChain splitters to preserve document structure while respecting token limits.

### HTML Header-Based Splitting

For HTML content, the system uses **`HTMLHeaderTextSplitter`** (lines 665-672), configured to split on `<h1>`, `<h2>`, and `<h3>` tags. This approach maintains semantic boundaries defined by the document's header hierarchy, keeping related content together in single chunks where possible.

### Markdown Structure Preservation

Markdown files trigger **`MarkdownHeaderTextSplitter`** (lines 675-682), which respects `#`, `##`, and `###` headers. This strategy ensures that logical sections remain intact during the embedding preparation process, preserving the document's organizational intent.

### Recursive Character Splitting for Plain Text

For plain text or unclassified content, the system falls back to **`RecursiveCharacterTextSplitter`** (lines 688-695). This splitter uses the configured `CHUNK_SIZE`, `CHUNK_OVERLAP`, and a token-count length function from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py) to create uniform chunks without structural dependencies.

## Secondary Chunking and Quality Filtering

The pipeline includes safety mechanisms to handle edge cases where primary splitters produce oversized or trivial chunks.

### Oversized Chunk Handling

The **`_apply_secondary_chunking()`** function (lines 398-415) processes HTML and Markdown outputs. When semantic splitters create chunks exceeding `CHUNK_SIZE`, this function re-splits the oversized content using the plain-text recursive splitter, ensuring no chunk violates token limits.

### Minimum Size Filtering

Before returning results, the system filters chunks in the final output stage (lines 782-892). Chunks smaller than `MIN_CHUNK_SIZE` (5 tokens by default) are dropped unless they represent the sole content chunk. This prevents useless embeddings from whitespace or fragmentary text while preserving critical short content.

## Practical Implementation Example

The **`chunk_text()`** function (lines 418-495) orchestrates the entire pipeline. Here are practical usage patterns:

```python
from open_notebook.utils.chunking import chunk_text, ContentType

# Example 1 – plain text with automatic detection

plain = "Lorem ipsum " * 200   # roughly 300 tokens

plain_chunks = chunk_text(plain)
print(f"Plain chunks: {len(plain_chunks)}")

# Example 2 – Markdown file with extension detection

md_path = "docs/overview.md"
with open(md_path) as f:
    md_text = f.read()
md_chunks = chunk_text(md_text, file_path=md_path)
print(f"Markdown chunks: {len(md_chunks)}")

# Example 3 – Force HTML splitter for raw web pages

html = "<html><body><h1>Title</h1><p>Long paragraph …</p></body></html>"
html_chunks = chunk_text(html, content_type=ContentType.HTML)
print(f"HTML chunks: {len(html_chunks)}")

```

## Summary

- **Configurable token boundaries**: The system uses environment variables to set chunk size (400), overlap (15%), and minimum size (5), with strict validation in `_get_chunk_size()`, `_get_chunk_overlap()`, and `_get_min_chunk_size()`.
- **Intelligent content detection**: Hybrid logic in `detect_content_type()` combines extension mapping and heuristic analysis (0.8 confidence threshold) to correctly classify HTML, Markdown, and plain text.
- **Structure-aware splitting**: Specialized LangChain splitters preserve semantic boundaries—`HTMLHeaderTextSplitter` for `<h1>-<h3>`, `MarkdownHeaderTextSplitter` for headers, and `RecursiveCharacterTextSplitter` for unstructured text.
- **Robust size enforcement**: Secondary chunking via `_apply_secondary_chunking()` and minimum-size filtering ensure all output chunks meet token limits while avoiding embedding noise.

## Frequently Asked Questions

### What is the default chunk size in Open Notebook?

The default chunk size is **400 tokens**, with a **15% overlap** between consecutive chunks and a **5-token minimum** threshold. These values are hardcoded as fallbacks in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) but can be overridden via environment variables read by `_get_chunk_size()`, `_get_chunk_overlap()`, and `_get_min_chunk_size()`.

### How does the system handle files with incorrect extensions?

When the extension is generic or missing, `detect_content_type_from_heuristics()` analyzes content samples to calculate HTML and Markdown confidence scores. If either score exceeds **0.8**, the system overrides the extension-based classification, ensuring proper splitting strategy selection even for mislabeled files or raw web content pasted as text.

### What happens if a semantic chunk exceeds the token limit?

The **`_apply_secondary_chunking()`** function (lines 398-415) catches oversized chunks from HTML or Markdown splitters and reprocesses them using the recursive plain-text splitter. This ensures no chunk exceeds the configured `CHUNK_SIZE` while preserving as much semantic structure as possible.

### Where is the core chunking logic implemented?

All chunking functionality resides in **[`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py)**, with token counting utilities imported from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py). The module depends on `langchain-text-splitters` for the `HTMLHeaderTextSplitter`, `MarkdownHeaderTextSplitter`, and `RecursiveCharacterTextSplitter` implementations.