Content Extraction Pipeline Using the content-core Library in Open Notebook

Open Notebook's ingestion workflow uses a LangGraph state machine in open_notebook/graphs/source.py that orchestrates content_core.extract_content() to convert URLs and files into clean markdown, handling errors through sentinel detection before persisting to the Source model.

The lfnovo/open-notebook repository implements a robust, async-first content extraction pipeline using the content-core library to transform unstructured data from web pages, PDFs, videos, and audio files into structured markdown. This pipeline centers on a LangGraph state machine that coordinates the extraction lifecycle from initial state preparation through error handling and persistence.

Architecture of the Content Extraction Pipeline

The pipeline is built as a compiled LangGraph workflow (source_graph) that processes Source entities through a series of async nodes. Each node manages a specific phase of extraction, ensuring type-specific handlers are invoked while maintaining a consistent state interface.

LangGraph State Machine

The extraction logic resides in open_notebook/graphs/source.py, which defines a stateful graph managing the ProcessSourceState imported from content_core.common. This state dictionary tracks the target URL or file path, desired output format, and optional speech-to-text configuration. The graph compiles into a runnable object invoked by the Sources Service when users create new sources.

ProcessSourceState Configuration

Before extraction begins, the pipeline constructs a ProcessSourceState dictionary that instructs the content-core library how to process the input. This state includes the url or file reference, the output_format (typically "markdown" or "text"), and optional audio handling parameters. According to the source code in open_notebook/graphs/source.py, the state preparation occurs between lines 34-61, where the graph initializes the extraction context.

Step-by-Step Extraction Workflow

The content extraction pipeline using the content-core library follows six distinct phases:

1. Speech-to-Text Model Selection

For audio and video inputs, the pipeline first resolves the default transcription model using ModelManager. The graph injects audio_provider and audio_model into the state (lines 62-74 in open_notebook/graphs/source.py), ensuring content-core calls the correct backend service when processing multimedia files or YouTube URLs.

2. Core Extraction via content-core

The central operation is an async call to extract_content(state) from the content_core package (lines 78-80). This function abstracts file-type-specific logic:

• Web pages: Fetches HTML, strips boilerplate, and returns clean markdown
• PDFs and Office files: Uses backends like pdfminer to extract text
• Images: Applies pytesseract for OCR
• Audio and video: Processes through whisper or similar transcription services
• YouTube URLs: Extracts subtitles or downloads audio for transcription when a speech-to-text model is configured

3. Error Sentinel Detection

The pipeline implements soft-failure detection by checking the extraction result. When content_core encounters an error, it returns a state with title="Error" and a content string beginning with "Failed to extract content:". The graph detects this sentinel between lines 80-90 in open_notebook/graphs/source.py and raises a ValueError, marking the source as retryable rather than persisting corrupted data.

4. Result Persistence

Upon successful extraction, the save_source node writes the processed text to a Source record (lines 10-31). This node attaches the extracted Asset containing the URL or file path, updates the source title if the extraction provided one, and optionally initiates vector embedding for downstream retrieval.

5. Optional Content Transformation

If the user requested post-processing, the graph branches to the transform_content node (lines 43-59). This node runs additional LangGraph transformations—such as summarization or keyword extraction—on the freshly extracted text before final storage.

Supported File Types and Formats

The content-core library (locked at version 1.14.1 in uv.lock) provides unified handling for diverse formats:

• Structured text: HTML, Markdown, plain text
• Documents: PDF, Microsoft Office formats
• Multimedia: MP3, MP4, WAV, common image formats
• Web content: Direct URLs and YouTube links

Each format routes through specialized backends while exposing a consistent ProcessSourceState interface, allowing the Open Notebook graph to remain agnostic to underlying extraction implementations.

Implementation Examples

Running Extraction Manually

For debugging or custom scripts, you can invoke the content-core extractor directly:

from content_core import extract_content
from content_core.common import ProcessSourceState

# Build the state expected by content-core

state: ProcessSourceState = {
    "url": "https://example.com/article.html",
    "output_format": "markdown",   # could also be "text"

    # optional audio handling – will be filled in by Open Notebook if a model is set

}

# Perform the extraction (async)

processed = await extract_content(state)

print("Title:", processed.title)
print("Extracted markdown:", processed.content[:500])

Graph Node Implementation

The internal graph node in open_notebook/graphs/source.py wraps the extraction call:

async def content_process(state: SourceState) -> dict:
    # Configure content_state and add speech-to-text model

    processed_state = await extract_content(state["content_state"])
    # Error-sentinel handling omitted for brevity

    return {"content_state": processed_state}

Service Layer Invocation

Create sources through the public API service for production use:

from open_notebook.services import SourcesService

svc = SourcesService()
result = svc.create_source(
    notebooks=["notebook:123"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
    async_processing=True,
)

# Returns a SourceProcessingResult containing the command_id to poll later

print("Job submitted:", result.command_id)

Summary

• The extraction pipeline centers on open_notebook/graphs/source.py, which implements a LangGraph state machine coordinating the content extraction workflow.
• content-core provides the extract_content() function and ProcessSourceState interface, handling web pages, PDFs, Office files, images, audio, and video through unified async calls.
• Error detection relies on sentinel values (title="Error") to trigger retry logic rather than persisting failures.
• The pipeline integrates with ModelManager to resolve speech-to-text models for audio transcription.
• Results persist through the Sources Service layer, with optional post-processing transformations available before final storage.

Frequently Asked Questions

What is the content-core library in Open Notebook?

The content-core library is a dependency (version 1.14.1) that provides the extract_content() function and ProcessSourceState type used by Open Notebook's extraction pipeline. It abstracts file-type-specific logic, allowing the application to extract clean markdown from web pages, documents, and multimedia files through a single async interface.

How does Open Notebook handle extraction errors?

The pipeline detects soft failures by checking if the content_core response contains title="Error" and content starting with "Failed to extract content:". When this sentinel is detected in open_notebook/graphs/source.py (lines 80-90), the graph raises a ValueError to mark the source as retryable rather than storing the error message as a valid document.

Can the pipeline process YouTube videos?

Yes. When processing YouTube URLs, the content-core library either extracts available subtitles or, when a default speech-to-text model is configured through ModelManager, downloads the audio stream and runs transcription via the configured provider.

Where is the extraction graph invoked in the codebase?

The compiled source_graph is invoked by the Sources Service, typically defined in api/sources_service.py. This service creates the ProcessSourceState, kicks off the LangGraph workflow, and returns either the completed Source object or a SourceProcessingResult containing a command ID for async job polling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}