How Open Notebook's Content Extraction Pipeline Uses the content-core Library

Open Notebook implements a LangGraph-based content extraction pipeline that delegates document processing to the content-core library's extract_content function, handling web pages, PDFs, audio, and video through a unified async interface defined in open_notebook/graphs/source.py.

The lfnovo/open-notebook repository relies on the content-core library (locked at version 1.14.1 in uv.lock) as its primary extraction engine within a state-driven ingestion workflow. This pipeline transforms raw URLs and file uploads into clean markdown or text, orchestrating format detection, speech-to-text transcription, and error recovery before persisting the results for downstream AI workflows.

Architecture of the Extraction Pipeline

The content extraction pipeline operates as a compiled LangGraph state machine (source_graph) that manages the lifecycle of a Source object from ingestion to storage. The workflow is defined in open_notebook/graphs/source.py and interacts with the content-core library through a standardized state dictionary called ProcessSourceState.

At its core, the pipeline abstracts complex document handling—whether fetching web HTML, parsing PDFs with pdfminer, transcribing audio via whisper, or extracting text from images using pytesseract—behind a single async function call provided by the external library.

Step-by-Step Pipeline Execution

Preparing the ProcessSourceState

The pipeline begins by constructing a ProcessSourceState dictionary, imported from content_core.common. This state object specifies the target URL or file path, the desired output format (typically "markdown" or "text"), and optional parameters for audio processing.

In open_notebook/graphs/source.py, the graph initializes this state to instruct the extractor which resource to process and how to format the results.

Configuring Speech-to-Text Models

Before invoking the extractor, the pipeline checks for audio or video content that requires transcription. The system queries the default speech-to-text model via ModelManager and injects audio_provider and audio_model values into the state. This configuration ensures that content-core routes audio transcription requests to the correct backend service without requiring manual provider selection.

Executing the Core Extraction

The central operation of the pipeline is a single async call to extract_content(state) from the content_core package. This function handles the actual content extraction:

• Web pages: Fetches HTML, strips boilerplate, and returns clean markdown.
• Documents: Processes PDFs, Office files, and images using appropriate backends.
• Media: Transcodes audio and video files to text using the configured speech-to-text model.
• YouTube URLs: Either extracts existing subtitles or downloads audio for transcription when a speech-to-text model is configured.

Handling Extraction Failures

The content-core library signals soft failures by returning a state where title="Error" and the content string begins with "Failed to extract content:". The graph detects this sentinel in open_notebook/graphs/source.py and raises a ValueError, marking the source as retryable rather than storing a corrupted document. This sentinel-based approach allows the pipeline to distinguish between temporary extraction errors and successfully processed content.

Persisting Results and Optional Transformations

Upon successful extraction, the save_source node writes the processed text to a Source record, attaches the extracted Asset containing the URL or file path, and updates the document title if the extraction provided one. The pipeline then optionally triggers vector embedding for semantic search.

If the user requested post-processing (such as summarization or keyword extraction), the graph branches to the transform_content node, which applies LangGraph transformations to the freshly extracted text before final storage.

Integration with the Sources Service

The compiled source_graph is invoked by the Sources Service defined in api/sources_service.py. When a user creates a new source via the API, the service instantiates the ProcessSourceState, kicks off the graph execution, and returns either the completed Source object or a SourceProcessingResult containing a command_id for async job polling.

This separation of concerns allows the service layer to handle HTTP request validation while delegating the complex extraction logic to the state machine and content-core library.

Practical Implementation Examples

Running Extraction Manually

For debugging or custom scripts, you can invoke the content-core extractor directly:

from content_core import extract_content
from content_core.common import ProcessSourceState

# Build the state expected by content-core

state: ProcessSourceState = {
    "url": "https://example.com/article.html",
    "output_format": "markdown",   # could also be "text"

    # optional audio handling – will be filled in by Open Notebook if a model is set

}

# Perform the extraction (async)

processed = await extract_content(state)

print("Title:", processed.title)
print("Extracted markdown:", processed.content[:500])

Graph Node Implementation

The content_process node within the graph demonstrates how Open Notebook wraps the library call:

# Inside open_notebook/graphs/source.py

async def content_process(state: SourceState) -> dict:
    # … configure content_state, add speech-to-text model …

    processed_state = await extract_content(state["content_state"])
    # … error-sentinel handling omitted …

    return {"content_state": processed_state}

Creating Sources via the Service Layer

To trigger the full pipeline through the public API:

from open_notebook.services import SourcesService

svc = SourcesService()
result = svc.create_source(
    notebooks=["notebook:123"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
    async_processing=True,
)

# result is a SourceProcessingResult containing the command_id to poll later

print("Job submitted:", result.command_id)

Summary

• Centralized Extraction: The content-core library provides the extract_content function that handles all document formats through a single async interface.
• State-Driven Architecture: The pipeline uses a LangGraph state machine in open_notebook/graphs/source.py to manage ProcessSourceState through preparation, extraction, and persistence.
• Error Sentinel Pattern: Failed extractions return title="Error" and a specific content prefix, allowing the graph to raise exceptions and mark jobs as retryable.
• Service Integration: The SourcesService in api/sources_service.py bridges user requests to the compiled source_graph, supporting both synchronous and asynchronous processing modes.
• Version Lock: The dependency is pinned to content_core==1.14.1 in uv.lock, ensuring consistent extraction behavior across deployments.

Frequently Asked Questions

What is the role of the ProcessSourceState dictionary?

The ProcessSourceState is a typed dictionary imported from content_core.common that configures the extraction job. It specifies the target URL or file, the desired output format (markdown or text), and optional audio provider settings for speech-to-text processing. This state object is the primary contract between Open Notebook and the content-core library.

How does the pipeline handle YouTube videos and audio files?

When processing YouTube URLs or audio uploads, the pipeline checks for a configured speech-to-text model via ModelManager and injects the audio_provider and audio_model into the state. The content-core library then either extracts existing subtitles or downloads the audio track and transcribes it using the specified model, returning the transcript as markdown content.

What happens when content-core fails to extract content?

The library signals failures by returning a state where title="Error" and the content string starts with "Failed to extract content:". The graph detects this sentinel pattern in open_notebook/graphs/source.py and raises a ValueError, which marks the source as retryable rather than persisting the error as a valid document. This prevents corrupted data from entering the vector store.

Where is the extraction pipeline entry point for API requests?

The entry point is the SourcesService class in api/sources_service.py. This service creates the initial Source record, builds the ProcessSourceState, and invokes the compiled source_graph. For async processing, it returns a SourceProcessingResult with a command_id that clients can poll to track extraction progress.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}