# How Open Notebook's Content Extraction Pipeline Uses the content-core Library

> Discover how Open Notebook's LangGraph pipeline employs the content-core library for efficient content extraction from diverse sources like web pages, PDFs, audio, and video via a unified async interface.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-22

---

**Open Notebook implements a LangGraph-based content extraction pipeline that delegates document processing to the `content-core` library's `extract_content` function, handling web pages, PDFs, audio, and video through a unified async interface defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py).**

The `lfnovo/open-notebook` repository relies on the `content-core` library (locked at version `1.14.1` in `uv.lock`) as its primary extraction engine within a state-driven ingestion workflow. This pipeline transforms raw URLs and file uploads into clean markdown or text, orchestrating format detection, speech-to-text transcription, and error recovery before persisting the results for downstream AI workflows.

## Architecture of the Extraction Pipeline

The content extraction pipeline operates as a compiled LangGraph state machine (`source_graph`) that manages the lifecycle of a `Source` object from ingestion to storage. The workflow is defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and interacts with the `content-core` library through a standardized state dictionary called `ProcessSourceState`.

At its core, the pipeline abstracts complex document handling—whether fetching web HTML, parsing PDFs with `pdfminer`, transcribing audio via `whisper`, or extracting text from images using `pytesseract`—behind a single async function call provided by the external library.

## Step-by-Step Pipeline Execution

### Preparing the ProcessSourceState

The pipeline begins by constructing a `ProcessSourceState` dictionary, imported from `content_core.common`. This state object specifies the target URL or file path, the desired output format (typically `"markdown"` or `"text"`), and optional parameters for audio processing.

In [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), the graph initializes this state to instruct the extractor which resource to process and how to format the results.

### Configuring Speech-to-Text Models

Before invoking the extractor, the pipeline checks for audio or video content that requires transcription. The system queries the default speech-to-text model via `ModelManager` and injects `audio_provider` and `audio_model` values into the state. This configuration ensures that `content-core` routes audio transcription requests to the correct backend service without requiring manual provider selection.

### Executing the Core Extraction

The central operation of the pipeline is a single async call to `extract_content(state)` from the `content_core` package. This function handles the actual content extraction:

- **Web pages**: Fetches HTML, strips boilerplate, and returns clean markdown.
- **Documents**: Processes PDFs, Office files, and images using appropriate backends.
- **Media**: Transcodes audio and video files to text using the configured speech-to-text model.
- **YouTube URLs**: Either extracts existing subtitles or downloads audio for transcription when a speech-to-text model is configured.

### Handling Extraction Failures

The `content-core` library signals soft failures by returning a state where `title="Error"` and the content string begins with `"Failed to extract content:"`. The graph detects this sentinel in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and raises a `ValueError`, marking the source as *retryable* rather than storing a corrupted document. This sentinel-based approach allows the pipeline to distinguish between temporary extraction errors and successfully processed content.

### Persisting Results and Optional Transformations

Upon successful extraction, the `save_source` node writes the processed text to a `Source` record, attaches the extracted `Asset` containing the URL or file path, and updates the document title if the extraction provided one. The pipeline then optionally triggers vector embedding for semantic search.

If the user requested post-processing (such as summarization or keyword extraction), the graph branches to the `transform_content` node, which applies LangGraph transformations to the freshly extracted text before final storage.

## Integration with the Sources Service

The compiled `source_graph` is invoked by the **Sources Service** defined in [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py). When a user creates a new source via the API, the service instantiates the `ProcessSourceState`, kicks off the graph execution, and returns either the completed `Source` object or a `SourceProcessingResult` containing a `command_id` for async job polling.

This separation of concerns allows the service layer to handle HTTP request validation while delegating the complex extraction logic to the state machine and `content-core` library.

## Practical Implementation Examples

### Running Extraction Manually

For debugging or custom scripts, you can invoke the `content-core` extractor directly:

```python
from content_core import extract_content
from content_core.common import ProcessSourceState

# Build the state expected by content-core

state: ProcessSourceState = {
    "url": "https://example.com/article.html",
    "output_format": "markdown",   # could also be "text"

    # optional audio handling – will be filled in by Open Notebook if a model is set

}

# Perform the extraction (async)

processed = await extract_content(state)

print("Title:", processed.title)
print("Extracted markdown:", processed.content[:500])

```

### Graph Node Implementation

The `content_process` node within the graph demonstrates how Open Notebook wraps the library call:

```python

# Inside open_notebook/graphs/source.py

async def content_process(state: SourceState) -> dict:
    # … configure content_state, add speech-to-text model …

    processed_state = await extract_content(state["content_state"])
    # … error-sentinel handling omitted …

    return {"content_state": processed_state}

```

### Creating Sources via the Service Layer

To trigger the full pipeline through the public API:

```python
from open_notebook.services import SourcesService

svc = SourcesService()
result = svc.create_source(
    notebooks=["notebook:123"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
    async_processing=True,
)

# result is a SourceProcessingResult containing the command_id to poll later

print("Job submitted:", result.command_id)

```

## Summary

- **Centralized Extraction**: The `content-core` library provides the `extract_content` function that handles all document formats through a single async interface.
- **State-Driven Architecture**: The pipeline uses a LangGraph state machine in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) to manage `ProcessSourceState` through preparation, extraction, and persistence.
- **Error Sentinel Pattern**: Failed extractions return `title="Error"` and a specific content prefix, allowing the graph to raise exceptions and mark jobs as retryable.
- **Service Integration**: The `SourcesService` in [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py) bridges user requests to the compiled `source_graph`, supporting both synchronous and asynchronous processing modes.
- **Version Lock**: The dependency is pinned to `content_core==1.14.1` in `uv.lock`, ensuring consistent extraction behavior across deployments.

## Frequently Asked Questions

### What is the role of the `ProcessSourceState` dictionary?

The `ProcessSourceState` is a typed dictionary imported from `content_core.common` that configures the extraction job. It specifies the target URL or file, the desired output format (markdown or text), and optional audio provider settings for speech-to-text processing. This state object is the primary contract between Open Notebook and the `content-core` library.

### How does the pipeline handle YouTube videos and audio files?

When processing YouTube URLs or audio uploads, the pipeline checks for a configured speech-to-text model via `ModelManager` and injects the `audio_provider` and `audio_model` into the state. The `content-core` library then either extracts existing subtitles or downloads the audio track and transcribes it using the specified model, returning the transcript as markdown content.

### What happens when `content-core` fails to extract content?

The library signals failures by returning a state where `title="Error"` and the content string starts with `"Failed to extract content:"`. The graph detects this sentinel pattern in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and raises a `ValueError`, which marks the source as retryable rather than persisting the error as a valid document. This prevents corrupted data from entering the vector store.

### Where is the extraction pipeline entry point for API requests?

The entry point is the `SourcesService` class in [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py). This service creates the initial `Source` record, builds the `ProcessSourceState`, and invokes the compiled `source_graph`. For async processing, it returns a `SourceProcessingResult` with a `command_id` that clients can poll to track extraction progress.