# Open-Notebook Source Ingestion Workflow: Extract, Embed, and Save Explained

> Explore the Open-Notebook source ingestion workflow. Learn how to extract content, generate vector embeddings, and save data using LangGraph, content-core, Surreal-Commands, and SurrealDB.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Open-Notebook processes documents through a three-stage LangGraph pipeline that extracts content using content-core, generates vector embeddings asynchronously via Surreal-Commands, and persists results to the SurrealDB database.**

The source ingestion workflow in [lfnovo/open-notebook](https://github.com/lfnovo/open-notebook) transforms raw documents—PDFs, webpages, YouTube videos, and more—into searchable, semantically-rich data. This pipeline is orchestrated as a state machine using LangGraph, ensuring each stage completes successfully before triggering the next, while long-running operations like embedding happen in the background.

## The Three-Stage Pipeline Overview

Open-Notebook implements a structured **extract → embed → save** workflow that handles content processing without blocking the API. The pipeline lives in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and processes documents through distinct nodes that manage extraction, vectorization, and persistence.

- **Extract**: Raw files or URLs are processed by content extraction engines
- **Embed**: Text is chunked and vectorized for semantic search
- **Save**: Processed content is persisted to the database with optional transformations

## Stage 1: Content Extraction

The ingestion workflow begins at the `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) (line 78). This node invokes the **`content-core`** library to handle format-specific parsing.

When a user uploads a document or provides a URL, the system initializes a `ProcessSourceState` containing the source metadata. The `extract_content` function (called at line 78) automatically selects the appropriate engine:

- **HTML scraper** for webpages
- **PDF parser** for documents
- **YouTube transcript fetcher** for videos

The extraction result populates the state with `title`, `content`, `url`, `file_path`, and other metadata fields. Default configurations for these engines are set between lines 35-60 in the same file, allowing customization of output formats like Markdown.

## Stage 2: Vector Embedding

Once text is extracted, the workflow optionally triggers the **embedding stage** to enable semantic search. Unlike extraction, this stage runs **asynchronously** to prevent API timeouts during large document processing.

The `Source.vectorize` method in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) (lines 77-95) submits a background job via `submit_command("embed_source", …)`. This Surreal-Commands job:

1. Chunks the extracted text into manageable segments
2. Generates embeddings using the unified **Esperanto embedding service**
3. Bulk-writes `source_embedding` records to the database

This fire-and-forget approach means the `save_source` node can complete immediately while vectorization continues in the background. The actual embedding logic resides in the command implementation under the `commands/` directory.

## Stage 3: Saving and Persistence

The `save_source` node (lines 10-28 in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)) persists the processed content to the SurrealDB `source` table. This node:

- Loads the existing `Source` record by ID
- Updates the `asset` field with the URL or file path
- Stores the extracted `title` and `full_text` content
- Calls `await source.save()` to commit changes

If the `embed` flag is set to `true` (checked at lines 31-38), the node triggers the asynchronous vectorization job before completing. This ensures the database contains the full text immediately, while search indexing happens concurrently.

## Optional Stage 4: Transformations

After saving, the workflow can apply user-selected **transformations** such as summarization or keyword extraction. The `trigger_transformations` function (lines 44-49) checks for requested transformations and conditionally routes to the `transform_content` node (lines 62-71).

Each transformation runs its own LangGraph defined in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py), creating `SourceInsight` records that are stored alongside the original source. These insights appear in the UI as processed metadata derived from the content.

## Orchestrating the Workflow with LangGraph

The entire source ingestion workflow is defined as a compiled state graph in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py):

```python
from langgraph.graph import StateGraph, START, END

workflow = StateGraph(SourceState)
workflow.add_node("content_process", content_process)   # Stage 1: Extract

workflow.add_node("save_source", save_source)           # Stage 3: Save

workflow.add_node("transform_content", transform_content)  # Stage 4: Transform

workflow.add_edge(START, "content_process")
workflow.add_edge("content_process", "save_source")
workflow.add_conditional_edges("save_source", trigger_transformations, ["transform_content"])
workflow.add_edge("transform_content", END)

source_graph = workflow.compile()

```

This state machine ensures that extraction must complete before saving, and transformations only run after successful persistence.

## Invoking the Pipeline

### Manual Invocation via Python

You can trigger the workflow directly from Python for custom ingestion scripts:

```python
from open_notebook.graphs.source import source_graph
from open_notebook.domain.notebook import Source
from open_notebook.domain.transformation import Transformation

async def ingest_source(source_id: str, notebook_ids: list[str], embed: bool = True):
    # Load the source record

    source = await Source.get(source_id)
    
    # Prepare content state for extraction

    content_state = {
        "url": source.asset.url,
        "file_path": source.asset.file_path,
        "url_engine": "auto",
        "document_engine": "auto",
        "output_format": "markdown",
    }
    
    # Load selected transformations

    transformations = [await Transformation.get(tid) for tid in ["summarize", "keywords"]]
    
    # Execute the graph

    result = await source_graph.ainvoke({
        "content_state": content_state,
        "notebook_ids": notebook_ids,
        "apply_transformations": transformations,
        "embed": embed,
        "source_id": source_id,
    })
    
    return result["source"]

```

### HTTP API Trigger

The UI typically invokes the workflow through the `process_source` command in [`commands/source_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/source_commands.py):

```bash
curl -X POST http://localhost:5055/commands/process_source \
    -H "Content-Type: application/json" \
    -d '{
          "source_id": "source:123",
          "content_state": {"url": "https://example.com/article"},
          "notebook_ids": ["notebook:45"],
          "transformations": ["summarize"],
          "embed": true
        }'

```

The command returns immediate confirmation while processing continues asynchronously:

```json
{
  "success": true,
  "source_id": "source:123",
  "embedded_chunks": 0,
  "insights_created": 1,
  "processing_time": 2.34
}

```

Note that `embedded_chunks` returns `0` initially because embedding runs as a background job. The count updates once the Surreal-Commands job completes.

## Summary

- **Content extraction** uses the `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) to parse PDFs, HTML, and videos via the content-core library
- **Vector embedding** runs asynchronously through `Source.vectorize` in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py), submitting Surreal-Commands jobs that chunk text and generate embeddings via the Esperanto service
- **Persistence** occurs in the `save_source` node, which updates the `source` table with extracted metadata and triggers optional background embedding
- **Transformations** create `SourceInsight` records after saving, enabling summarization and keyword extraction without blocking the main workflow
- The entire pipeline is orchestrated by a LangGraph state machine defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)

## Frequently Asked Questions

### What file formats does the extraction stage support?

The extraction stage in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) supports multiple formats through the content-core library. The `extract_content` function automatically selects HTML scrapers for webpages, PDF parsers for documents, and YouTube transcript fetchers for video URLs. The `url_engine` and `document_engine` parameters in the `content_state` allow customization of these parsers, with "auto" enabling automatic format detection.

### How does the system handle embedding failures for large documents?

Because embedding runs asynchronously via Surreal-Commands, failures are handled by the command queue rather than the main ingestion workflow. The `Source.vectorize` method submits a background job that chunks text before embedding, preventing memory issues. You can check the processing status later using `await source.get_status()` or `await source.get_processing_progress()` to see if the embedding succeeded or retry failed chunks.

### Can I skip the embedding stage to speed up ingestion?

Yes, the embedding stage is optional. When invoking the workflow via the `process_source` command or direct graph invocation, set the `embed` parameter to `false`. The `save_source` node checks this flag at lines 31-38 in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and only calls `source.vectorize()` when explicitly requested. The source will still be saved with full-text content but will not support semantic search until manually vectorized later.

### Where is the workflow state defined and how does it persist between stages?

The workflow state is defined by the `SourceState` and `ProcessSourceState` classes passed between nodes in the LangGraph. While the graph runs in memory, critical state is persisted to SurrealDB at each stage. The `save_source` node explicitly writes to the `source` table, and the `content_process` node updates the state object with extracted fields. This allows the workflow to resume or be tracked even if the process restarts, since the database serves as the source of truth.