Open-Notebook Source Ingestion Workflow: Extract, Embed, and Save Explained

Open-Notebook processes documents through a three-stage LangGraph pipeline that extracts content using content-core, generates vector embeddings asynchronously via Surreal-Commands, and persists results to the SurrealDB database.

The source ingestion workflow in lfnovo/open-notebook transforms raw documents—PDFs, webpages, YouTube videos, and more—into searchable, semantically-rich data. This pipeline is orchestrated as a state machine using LangGraph, ensuring each stage completes successfully before triggering the next, while long-running operations like embedding happen in the background.

The Three-Stage Pipeline Overview

Open-Notebook implements a structured extract → embed → save workflow that handles content processing without blocking the API. The pipeline lives in open_notebook/graphs/source.py and processes documents through distinct nodes that manage extraction, vectorization, and persistence.

• Extract: Raw files or URLs are processed by content extraction engines
• Embed: Text is chunked and vectorized for semantic search
• Save: Processed content is persisted to the database with optional transformations

Stage 1: Content Extraction

The ingestion workflow begins at the content_process node in open_notebook/graphs/source.py (line 78). This node invokes the content-core library to handle format-specific parsing.

When a user uploads a document or provides a URL, the system initializes a ProcessSourceState containing the source metadata. The extract_content function (called at line 78) automatically selects the appropriate engine:

• HTML scraper for webpages
• PDF parser for documents
• YouTube transcript fetcher for videos

The extraction result populates the state with title, content, url, file_path, and other metadata fields. Default configurations for these engines are set between lines 35-60 in the same file, allowing customization of output formats like Markdown.

Stage 2: Vector Embedding

Once text is extracted, the workflow optionally triggers the embedding stage to enable semantic search. Unlike extraction, this stage runs asynchronously to prevent API timeouts during large document processing.

The Source.vectorize method in open_notebook/domain/notebook.py (lines 77-95) submits a background job via submit_command("embed_source", …). This Surreal-Commands job:

1. Chunks the extracted text into manageable segments
2. Generates embeddings using the unified Esperanto embedding service
3. Bulk-writes source_embedding records to the database

This fire-and-forget approach means the save_source node can complete immediately while vectorization continues in the background. The actual embedding logic resides in the command implementation under the commands/ directory.

Stage 3: Saving and Persistence

The save_source node (lines 10-28 in open_notebook/graphs/source.py) persists the processed content to the SurrealDB source table. This node:

• Loads the existing Source record by ID
• Updates the asset field with the URL or file path
• Stores the extracted title and full_text content
• Calls await source.save() to commit changes

If the embed flag is set to true (checked at lines 31-38), the node triggers the asynchronous vectorization job before completing. This ensures the database contains the full text immediately, while search indexing happens concurrently.

Optional Stage 4: Transformations

After saving, the workflow can apply user-selected transformations such as summarization or keyword extraction. The trigger_transformations function (lines 44-49) checks for requested transformations and conditionally routes to the transform_content node (lines 62-71).

Each transformation runs its own LangGraph defined in open_notebook/graphs/transformation.py, creating SourceInsight records that are stored alongside the original source. These insights appear in the UI as processed metadata derived from the content.

Orchestrating the Workflow with LangGraph

The entire source ingestion workflow is defined as a compiled state graph in open_notebook/graphs/source.py:

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(SourceState)
workflow.add_node("content_process", content_process)   # Stage 1: Extract

workflow.add_node("save_source", save_source)           # Stage 3: Save

workflow.add_node("transform_content", transform_content)  # Stage 4: Transform

workflow.add_edge(START, "content_process")
workflow.add_edge("content_process", "save_source")
workflow.add_conditional_edges("save_source", trigger_transformations, ["transform_content"])
workflow.add_edge("transform_content", END)

source_graph = workflow.compile()

This state machine ensures that extraction must complete before saving, and transformations only run after successful persistence.

Invoking the Pipeline

Manual Invocation via Python

You can trigger the workflow directly from Python for custom ingestion scripts:

from open_notebook.graphs.source import source_graph
from open_notebook.domain.notebook import Source
from open_notebook.domain.transformation import Transformation

async def ingest_source(source_id: str, notebook_ids: list[str], embed: bool = True):
    # Load the source record

    source = await Source.get(source_id)
    
    # Prepare content state for extraction

    content_state = {
        "url": source.asset.url,
        "file_path": source.asset.file_path,
        "url_engine": "auto",
        "document_engine": "auto",
        "output_format": "markdown",
    }
    
    # Load selected transformations

    transformations = [await Transformation.get(tid) for tid in ["summarize", "keywords"]]
    
    # Execute the graph

    result = await source_graph.ainvoke({
        "content_state": content_state,
        "notebook_ids": notebook_ids,
        "apply_transformations": transformations,
        "embed": embed,
        "source_id": source_id,
    })
    
    return result["source"]

HTTP API Trigger

The UI typically invokes the workflow through the process_source command in commands/source_commands.py:

curl -X POST http://localhost:5055/commands/process_source \
    -H "Content-Type: application/json" \
    -d '{
          "source_id": "source:123",
          "content_state": {"url": "https://example.com/article"},
          "notebook_ids": ["notebook:45"],
          "transformations": ["summarize"],
          "embed": true
        }'

The command returns immediate confirmation while processing continues asynchronously:

{
  "success": true,
  "source_id": "source:123",
  "embedded_chunks": 0,
  "insights_created": 1,
  "processing_time": 2.34
}

Note that embedded_chunks returns 0 initially because embedding runs as a background job. The count updates once the Surreal-Commands job completes.

Summary

• Content extraction uses the content_process node in open_notebook/graphs/source.py to parse PDFs, HTML, and videos via the content-core library
• Vector embedding runs asynchronously through Source.vectorize in open_notebook/domain/notebook.py, submitting Surreal-Commands jobs that chunk text and generate embeddings via the Esperanto service
• Persistence occurs in the save_source node, which updates the source table with extracted metadata and triggers optional background embedding
• Transformations create SourceInsight records after saving, enabling summarization and keyword extraction without blocking the main workflow
• The entire pipeline is orchestrated by a LangGraph state machine defined in open_notebook/graphs/source.py

Frequently Asked Questions

What file formats does the extraction stage support?

The extraction stage in open_notebook/graphs/source.py supports multiple formats through the content-core library. The extract_content function automatically selects HTML scrapers for webpages, PDF parsers for documents, and YouTube transcript fetchers for video URLs. The url_engine and document_engine parameters in the content_state allow customization of these parsers, with "auto" enabling automatic format detection.

How does the system handle embedding failures for large documents?

Because embedding runs asynchronously via Surreal-Commands, failures are handled by the command queue rather than the main ingestion workflow. The Source.vectorize method submits a background job that chunks text before embedding, preventing memory issues. You can check the processing status later using await source.get_status() or await source.get_processing_progress() to see if the embedding succeeded or retry failed chunks.

Can I skip the embedding stage to speed up ingestion?

Yes, the embedding stage is optional. When invoking the workflow via the process_source command or direct graph invocation, set the embed parameter to false. The save_source node checks this flag at lines 31-38 in open_notebook/graphs/source.py and only calls source.vectorize() when explicitly requested. The source will still be saved with full-text content but will not support semantic search until manually vectorized later.

Where is the workflow state defined and how does it persist between stages?

The workflow state is defined by the SourceState and ProcessSourceState classes passed between nodes in the LangGraph. While the graph runs in memory, critical state is persisted to SurrealDB at each stage. The save_source node explicitly writes to the source table, and the content_process node updates the state object with extracted fields. This allows the workflow to resume or be tracked even if the process restarts, since the database serves as the source of truth.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}