# Handling Large File Processing in Open Notebook: Chunking and Embedding Strategies

> Learn how Open Notebook handles large file processing. Discover chunking and embedding strategies for efficient vector representation and API limit adherence.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-17

---

**Open Notebook handles large file processing by splitting documents into token-bounded chunks before embedding, then mean-pooling the results to create unified vector representations while respecting API limits.**

The lfnovo/open-notebook repository implements a robust pipeline for handling large file processing that prevents API payload limits and memory errors. By implementing intelligent content-aware chunking in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) and batched embedding with mean pooling in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), the system processes multi-megabyte documents without blocking the API.

## Chunking Strategy for Large Documents

The core chunking logic lives in **[`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py)**. This module implements content-type detection, configurable token budgets, and structure-aware splitting to handle PDFs, web pages, and markdown files of arbitrary size.

### Content-Type Detection

The **`detect_content_type()`** function first examines file extensions using the **`_EXTENSION_TO_CONTENT_TYPE`** mapping. When extensions are ambiguous or missing, it falls back to heuristics via **`_calculate_html_score`** and **`_calculate_markdown_score`** to determine whether the content is HTML, Markdown, or plain text.

### Configurable Token Budgets

Environment variables control the chunking parameters to match your embedding model's constraints:

- **`OPEN_NOTEBOOK_CHUNK_SIZE`** (default 400 tokens) — read by `_get_chunk_size()`【/blob/main/open_notebook/utils/chunking.py#L33-L52】
- **`OPEN_NOTEBOOK_CHUNK_OVERLAP`** (default ~15% of chunk size) — read by `_get_chunk_overlap()`【/blob/main/open_notebook/utils/chunking.py#L60-L78】
- **`OPEN_NOTEBOOK_MIN_CHUNK_SIZE`** (default 5 tokens) — filters degenerate fragments via `_get_min_chunk_size()`【/blob/main/open_notebook/utils/chunking.py#L88-L112】

### Splitting Logic and Fallbacks

The **`chunk_text()`** function selects splitters based on detected content type. HTML documents use `HTMLHeaderTextSplitter`, Markdown uses `MarkdownHeaderTextSplitter`, and plain text falls back to `RecursiveCharacterTextSplitter`. Large sections from HTML/Markdown undergo secondary chunking via **`_apply_secondary_chunking`** to enforce token limits【/blob/main/open_notebook/utils/chunking.py#L70-L74】. The function guarantees at least one non-empty chunk for any valid input【/blob/main/open_notebook/utils/chunking.py#L418-L495】.

```python
from open_notebook.utils.chunking import chunk_text

# Process a large markdown file

chunks = chunk_text(
    text=large_markdown_content,
    content_type="markdown",
    file_path="document.md"
)

```

## Embedding Pipeline for Chunked Content

The **[`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py)** module orchestrates the embedding generation for large files, automatically triggering chunking when token counts exceed configured limits.

### Token Counting and Decision Logic

The **`generate_embedding()`** function first checks token counts using **`token_count()`** from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py). If the text fits within `CHUNK_SIZE`, it calls the model directly; otherwise, it invokes `chunk_text()` to split the document before embedding【/blob/main/open_notebook/utils/embedding.py#L9-L75】.

### Batch Processing and Mean Pooling

Chunks are processed in batches respecting **`OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE`** (default 50), configured via **`_get_embedding_batch_size()`**【/blob/main/open_notebook/utils/embedding.py#L24-L42】. The **`generate_embeddings()`** function handles batching, while **`mean_pool_embeddings()`** aggregates chunk embeddings into a single vector representing the entire document【/blob/main/open_notebook/utils/embedding.py#L216-L275】.

```python
from open_notebook.utils.embedding import generate_embedding

# Automatically handles chunking for large inputs

embedding = await generate_embedding(
    text=very_long_document,
    model="text-embedding-3-small"
)

```

## End-to-End Processing Flow

The large file processing pipeline operates asynchronously to maintain API responsiveness:

1. **File upload** via the API's `sources` endpoint
2. **Text extraction** and passage to `generate_embedding()`
3. **Token validation** via `token_count()` to determine if chunking is required
4. **Conditional splitting** via `chunk_text()` with content-type detection
5. **Batched embedding** generation via `generate_embeddings()`
6. **Mean pooling** via `mean_pool_embeddings()` for unified storage in SurrealDB

## Configuration Reference

Tune the large file processing behavior using these environment variables:

| Variable | Default | Purpose |
|----------|---------|---------|
| `OPEN_NOTEBOOK_CHUNK_SIZE` | 400 tokens | Target tokens per chunk |
| `OPEN_NOTEBOOK_CHUNK_OVERLAP` | ~15% | Overlap between consecutive chunks |
| `OPEN_NOTEBOOK_MIN_CHUNK_SIZE` | 5 tokens | Minimum valid chunk size to filter debris |
| `OPEN_NOTEBOOK_EMBEDDING_BATCH_SIZE` | 50 | API requests per batch |

## Summary

- Open Notebook processes large files through token-based chunking in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py) and mean-pooled embeddings in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py).
- Content-aware splitters preserve document structure while enforcing token limits via environment variables.
- The `chunk_text()` function guarantees at least one non-empty chunk for any valid input, filtering degenerate fragments below `OPEN_NOTEBOOK_MIN_CHUNK_SIZE`.
- Mean pooling in `generate_embedding()` creates stable vector representations regardless of document length, simplifying storage in SurrealDB.
- Asynchronous processing ensures the API remains responsive during multi-megabyte file ingestion.

## Frequently Asked Questions

### How does Open Notebook handle PDF files that exceed API token limits?

According to the source code in [`open_notebook/utils/embedding.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/embedding.py), files are routed through `generate_embedding()`, which detects oversized content via `token_count()` from [`open_notebook/utils/token_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/token_utils.py). When content exceeds `CHUNK_SIZE`, the function automatically triggers `chunk_text()` to split the document into token-bounded chunks before embedding, then mean-pools the results into a single vector.

### What is the default chunk size and can it be configured?

The default chunk size is **400 tokens**, controlled by the `OPEN_NOTEBOOK_CHUNK_SIZE` environment variable in [`open_notebook/utils/chunking.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/chunking.py). You can adjust this value to match your specific embedding model's context window or API payload limits by setting the environment variable before application startup.

### How does the system maintain document coherence when splitting large files?

Open Notebook uses `HTMLHeaderTextSplitter` and `MarkdownHeaderTextSplitter` to preserve logical sections and headers during the initial split. If sections remain too large, the `_apply_secondary_chunking` function applies `RecursiveCharacterTextSplitter` to enforce token budgets while maintaining semantic boundaries where possible, discarding only fragments below `OPEN_NOTEBOOK_MIN_CHUNK_SIZE`.

### Why does Open Notebook use mean pooling for chunk embeddings?

Mean pooling via `mean_pool_embeddings()` creates a single fixed-length vector representing the entire document regardless of how many chunks were generated. This approach ensures consistent vector dimensions for storage in SurrealDB and guarantees that similarity search behaves uniformly across documents of varying lengths, whether they required splitting or not.