Handling Large File Processing in Open Notebook: Chunking and Embedding Strategies

Open Notebook handles large file processing by splitting documents into token-bounded chunks before embedding, then mean-pooling the results to create unified vector representations while respecting API limits.

The lfnovo/open-notebook repository implements a robust pipeline for handling large file processing that prevents API payload limits and memory errors. By implementing intelligent content-aware chunking in open_notebook/utils/chunking.py and batched embedding with mean pooling in open_notebook/utils/embedding.py, the system processes multi-megabyte documents without blocking the API.

Chunking Strategy for Large Documents

The core chunking logic lives in open_notebook/utils/chunking.py. This module implements content-type detection, configurable token budgets, and structure-aware splitting to handle PDFs, web pages, and markdown files of arbitrary size.

Content-Type Detection

The detect_content_type() function first examines file extensions using the _EXTENSION_TO_CONTENT_TYPE mapping. When extensions are ambiguous or missing, it falls back to heuristics via _calculate_html_score and _calculate_markdown_score to determine whether the content is HTML, Markdown, or plain text.

Configurable Token Budgets

Environment variables control the chunking parameters to match your embedding model's constraints:

• OPEN_NOTEBOOK_CHUNK_SIZE (default 400 tokens) — read by _get_chunk_size()【/blob/main/open_notebook/utils/chunking.py#L33-L52】
• OPEN_NOTEBOOK_CHUNK_OVERLAP (default ~15% of chunk size) — read by _get_chunk_overlap()【/blob/main/open_notebook/utils/chunking.py#L60-L78】
• OPEN_NOTEBOOK_MIN_CHUNK_SIZE (default 5 tokens) — filters degenerate fragments via _get_min_chunk_size()【/blob/main/open_notebook/utils/chunking.py#L88-L112】

Splitting Logic and Fallbacks

The chunk_text() function selects splitters based on detected content type. HTML documents use HTMLHeaderTextSplitter, Markdown uses MarkdownHeaderTextSplitter, and plain text falls back to RecursiveCharacterTextSplitter. Large sections from HTML/Markdown undergo secondary chunking via _apply_secondary_chunking to enforce token limits【/blob/main/open_notebook/utils/chunking.py#L70-L74】. The function guarantees at least one non-empty chunk for any valid input【/blob/main/open_notebook/utils/chunking.py#L418-L495】.

from open_notebook.utils.chunking import chunk_text

# Process a large markdown file

chunks = chunk_text(
    text=large_markdown_content,
    content_type="markdown",
    file_path="document.md"
)

Embedding Pipeline for Chunked Content

The open_notebook/utils/embedding.py module orchestrates the embedding generation for large files, automatically triggering chunking when token counts exceed configured limits.

Token Counting and Decision Logic

The generate_embedding() function first checks token counts using token_count() from open_notebook/utils/token_utils.py. If the text fits within CHUNK_SIZE, it calls the model directly; otherwise, it invokes chunk_text() to split the document before embedding【/blob/main/open_notebook/utils/embedding.py#L9-L75】.

Batch Processing and Mean Pooling

Chunks are processed in batches respecting OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE (default 50), configured via _get_embedding_batch_size()【/blob/main/open_notebook/utils/embedding.py#L24-L42】. The generate_embeddings() function handles batching, while mean_pool_embeddings() aggregates chunk embeddings into a single vector representing the entire document【/blob/main/open_notebook/utils/embedding.py#L216-L275】.

from open_notebook.utils.embedding import generate_embedding

# Automatically handles chunking for large inputs

embedding = await generate_embedding(
    text=very_long_document,
    model="text-embedding-3-small"
)

End-to-End Processing Flow

The large file processing pipeline operates asynchronously to maintain API responsiveness:

1. File upload via the API's sources endpoint
2. Text extraction and passage to generate_embedding()
3. Token validation via token_count() to determine if chunking is required
4. Conditional splitting via chunk_text() with content-type detection
5. Batched embedding generation via generate_embeddings()
6. Mean pooling via mean_pool_embeddings() for unified storage in SurrealDB

Configuration Reference

Tune the large file processing behavior using these environment variables:

Variable	Default	Purpose
OPEN_NOTEBOOK_CHUNK_SIZE	400 tokens	Target tokens per chunk
OPEN_NOTEBOOK_CHUNK_OVERLAP	~15%	Overlap between consecutive chunks
OPEN_NOTEBOOK_MIN_CHUNK_SIZE	5 tokens	Minimum valid chunk size to filter debris
OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE	50	API requests per batch

Summary

• Open Notebook processes large files through token-based chunking in open_notebook/utils/chunking.py and mean-pooled embeddings in open_notebook/utils/embedding.py.
• Content-aware splitters preserve document structure while enforcing token limits via environment variables.
• The chunk_text() function guarantees at least one non-empty chunk for any valid input, filtering degenerate fragments below OPEN_NOTEBOOK_MIN_CHUNK_SIZE.
• Mean pooling in generate_embedding() creates stable vector representations regardless of document length, simplifying storage in SurrealDB.
• Asynchronous processing ensures the API remains responsive during multi-megabyte file ingestion.

Frequently Asked Questions

How does Open Notebook handle PDF files that exceed API token limits?

According to the source code in open_notebook/utils/embedding.py, files are routed through generate_embedding(), which detects oversized content via token_count() from open_notebook/utils/token_utils.py. When content exceeds CHUNK_SIZE, the function automatically triggers chunk_text() to split the document into token-bounded chunks before embedding, then mean-pools the results into a single vector.

What is the default chunk size and can it be configured?

The default chunk size is 400 tokens, controlled by the OPEN_NOTEBOOK_CHUNK_SIZE environment variable in open_notebook/utils/chunking.py. You can adjust this value to match your specific embedding model's context window or API payload limits by setting the environment variable before application startup.

How does the system maintain document coherence when splitting large files?

Open Notebook uses HTMLHeaderTextSplitter and MarkdownHeaderTextSplitter to preserve logical sections and headers during the initial split. If sections remain too large, the _apply_secondary_chunking function applies RecursiveCharacterTextSplitter to enforce token budgets while maintaining semantic boundaries where possible, discarding only fragments below OPEN_NOTEBOOK_MIN_CHUNK_SIZE.

Why does Open Notebook use mean pooling for chunk embeddings?

Mean pooling via mean_pool_embeddings() creates a single fixed-length vector representing the entire document regardless of how many chunks were generated. This approach ensures consistent vector dimensions for storage in SurrealDB and guarantees that similarity search behaves uniformly across documents of varying lengths, whether they required splitting or not.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}