How Open-Notebook Extracts and Chunks Content from Various Sources

Open-Notebook uses a two-stage pipeline where the content_core library extracts raw text from sources like PDFs and YouTube videos, and a custom chunk_text utility splits that content into token-sized pieces using configurable size limits and content-aware boundaries.

The lfnovo/open-notebook repository handles diverse content types through a unified extraction and chunking system. This pipeline transforms everything from web pages to video transcripts into searchable, embeddable segments. Understanding how open-notebook extracts and chunks content reveals how the system maintains semantic coherence while optimizing for SurrealDB vector storage.

Content Extraction Pipeline

Orchestrating Extraction via the Source Graph

The entry point for all content ingestion is open_notebook/graphs/source.py. This module defines the extraction workflow that wraps the external content_core library. When processing begins, the graph receives a Source record containing metadata about the input—whether it is a PDF upload, a URL, or a YouTube link.

The graph calls extract_content from content_core, which returns a plain-text string. If extraction fails, the system generates user-friendly error messages such as "Could not extract content from this YouTube video" rather than raw stack traces.

Engine Selection and Configuration

The content_core library dynamically selects the appropriate extraction engine based on source type. Available engines include Docling, Firecrawl, Jina, and Simple, with defaults configured in open_notebook/domain/content_settings.py. This configuration file determines which engine handles specific MIME types or URL patterns, allowing the system to use specialized parsers for PDFs while falling back to simpler extractors for plain text.

Smart Text Chunking Strategy

Content-Type Detection

Once raw text is extracted, the system passes it to open_notebook/utils/chunking.py. The chunk_text function first determines whether the input is HTML, Markdown, or plain text. Detection happens automatically through content analysis, though you can force a specific type using the content_type argument or infer it from a file extension via the file_path parameter.

Token-Based Splitting with Guardrails

The chunking algorithm uses three critical environment variables to control output size:

• OPEN_NOTEBOOK_CHUNK_SIZE (default ~400 tokens)
• OPEN_NOTEBOOK_CHUNK_OVERLAP (controls context continuity between chunks)
• OPEN_NOTEBOOK_MIN_CHUNK_SIZE (default ~5 tokens, filters out fragments)

The implementation uses a language-agnostic tokenizer that handles English, CJK (Chinese, Japanese, Korean), and mixed-language strings. Chunks smaller than the minimum size are discarded, but the algorithm guarantees at least one chunk for any non-empty input.

Respecting Markdown Structure

For Markdown documents, the system applies a two-phase approach. Large documents are first split by section headers to preserve logical boundaries, then each section undergoes token-based chunking. This ensures that headers and their associated content remain semantically grouped while still fitting within the token window.

Embedding and Vector Storage

The open_notebook/utils/embedding.py module consumes the chunk list generated by the chunking utility. For short texts, it embeds a single chunk directly. For longer documents, it embeds each chunk individually and performs mean-pooling to average the vectors before storing them in SurrealDB.

This approach allows the system to represent lengthy documents as single dense vectors while maintaining the granularity needed for specific retrieval scenarios.

Implementation Examples

Extract content from a source using the graph wrapper:

from open_notebook.graphs.source import extract_content

# state is a ProcessSourceState object containing source metadata

raw_text = await extract_content(state)

Chunk the extracted text with automatic type detection:

from open_notebook.utils.chunking import chunk_text, ContentType

chunks = chunk_text(
    raw_text,
    content_type=ContentType.AUTO,  # Auto-detect HTML/Markdown/Plain

    file_path="my_document.pdf",    # Optional: aids type detection

)

Generate embeddings for storage:

from open_notebook.utils.embedding import embed_text

embedding_vector = await embed_text(
    text=raw_text,
    content_type=ContentType.AUTO,
    command_id="cmd-1234",  # For async job tracking

)

Summary

• Open-Notebook uses content_core via open_notebook/graphs/source.py to extract raw text from diverse sources including PDFs, URLs, and YouTube videos
• Engine selection defaults are managed in open_notebook/domain/content_settings.py, supporting Docling, Firecrawl, Jina, and Simple extractors
• The chunk_text function in open_notebook/utils/chunking.py detects content types automatically and splits text using token-based limits (~400 tokens default) with configurable overlap
• Markdown documents receive special treatment with header-aware splitting before token chunking to preserve semantic structure
• The embedding layer in open_notebook/utils/embedding.py applies mean-pooling for long documents before storage in SurrealDB

Frequently Asked Questions

What file types does Open-Notebook support for content extraction?

Open-Notebook supports PDFs, web pages, YouTube videos, and plain text files through the content_core library. The specific extractor engine (Docling for PDFs, Firecrawl for web pages, etc.) is selected automatically based on the source type and the configuration defined in open_notebook/domain/content_settings.py.

How does the chunking system handle multilingual text?

The chunk_text implementation uses a language-agnostic tokenizer that correctly segments English, CJK (Chinese, Japanese, Korean), and mixed-language strings. This ensures consistent chunk sizes regardless of character set, preventing scenarios where logographic languages produce artificially small chunks based on character count alone.

Why does Open-Notebook use mean-pooling for embeddings?

When processing long documents that exceed the single-chunk token limit, the system embeds each chunk separately and averages the resulting vectors through mean-pooling. This creates a single dense representation in SurrealDB that captures the full document context while maintaining the granular chunks needed for specific retrieval operations.

Can I customize the chunk size for specific documents?

Yes. While the default OPEN_NOTEBOOK_CHUNK_SIZE is approximately 400 tokens, you can pass the content_type parameter directly to chunk_text or configure the environment variables OPEN_NOTEBOOK_CHUNK_SIZE, OPEN_NOTEBOOK_CHUNK_OVERLAP, and OPEN_NOTEBOOK_MIN_CHUNK_SIZE to adjust the splitting behavior for your specific use case.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}