# Content Extraction Pipeline Using the content-core Library in Open Notebook

> Learn about Open Notebook's content extraction pipeline. This LangGraph state machine uses content-core to convert URLs and files to markdown, handling errors and persisting to the Source model.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-22

---

**Open Notebook's ingestion workflow uses a LangGraph state machine in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) that orchestrates `content_core.extract_content()` to convert URLs and files into clean markdown, handling errors through sentinel detection before persisting to the `Source` model.**

The lfnovo/open-notebook repository implements a robust, async-first content extraction pipeline using the content-core library to transform unstructured data from web pages, PDFs, videos, and audio files into structured markdown. This pipeline centers on a LangGraph state machine that coordinates the extraction lifecycle from initial state preparation through error handling and persistence.

## Architecture of the Content Extraction Pipeline

The pipeline is built as a compiled LangGraph workflow (`source_graph`) that processes **Source** entities through a series of async nodes. Each node manages a specific phase of extraction, ensuring type-specific handlers are invoked while maintaining a consistent state interface.

### LangGraph State Machine

The extraction logic resides in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), which defines a stateful graph managing the `ProcessSourceState` imported from `content_core.common`. This state dictionary tracks the target URL or file path, desired output format, and optional speech-to-text configuration. The graph compiles into a runnable object invoked by the **Sources Service** when users create new sources.

### ProcessSourceState Configuration

Before extraction begins, the pipeline constructs a `ProcessSourceState` dictionary that instructs the `content-core` library how to process the input. This state includes the `url` or file reference, the `output_format` (typically `"markdown"` or `"text"`), and optional audio handling parameters. According to the source code in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), the state preparation occurs between lines 34-61, where the graph initializes the extraction context.

## Step-by-Step Extraction Workflow

The content extraction pipeline using the content-core library follows six distinct phases:

### 1. Speech-to-Text Model Selection

For audio and video inputs, the pipeline first resolves the default transcription model using `ModelManager`. The graph injects `audio_provider` and `audio_model` into the state (lines 62-74 in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)), ensuring `content-core` calls the correct backend service when processing multimedia files or YouTube URLs.

### 2. Core Extraction via content-core

The central operation is an async call to `extract_content(state)` from the `content_core` package (lines 78-80). This function abstracts file-type-specific logic:

- **Web pages**: Fetches HTML, strips boilerplate, and returns clean markdown
- **PDFs and Office files**: Uses backends like `pdfminer` to extract text
- **Images**: Applies `pytesseract` for OCR
- **Audio and video**: Processes through `whisper` or similar transcription services
- **YouTube URLs**: Extracts subtitles or downloads audio for transcription when a speech-to-text model is configured

### 3. Error Sentinel Detection

The pipeline implements soft-failure detection by checking the extraction result. When `content_core` encounters an error, it returns a state with `title="Error"` and a `content` string beginning with `"Failed to extract content:"`. The graph detects this sentinel between lines 80-90 in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and raises a `ValueError`, marking the source as retryable rather than persisting corrupted data.

### 4. Result Persistence

Upon successful extraction, the `save_source` node writes the processed text to a `Source` record (lines 10-31). This node attaches the extracted `Asset` containing the URL or file path, updates the source title if the extraction provided one, and optionally initiates vector embedding for downstream retrieval.

### 5. Optional Content Transformation

If the user requested post-processing, the graph branches to the `transform_content` node (lines 43-59). This node runs additional LangGraph transformations—such as summarization or keyword extraction—on the freshly extracted text before final storage.

## Supported File Types and Formats

The `content-core` library (locked at version `1.14.1` in `uv.lock`) provides unified handling for diverse formats:

- **Structured text**: HTML, Markdown, plain text
- **Documents**: PDF, Microsoft Office formats
- **Multimedia**: MP3, MP4, WAV, common image formats
- **Web content**: Direct URLs and YouTube links

Each format routes through specialized backends while exposing a consistent `ProcessSourceState` interface, allowing the Open Notebook graph to remain agnostic to underlying extraction implementations.

## Implementation Examples

### Running Extraction Manually

For debugging or custom scripts, you can invoke the `content-core` extractor directly:

```python
from content_core import extract_content
from content_core.common import ProcessSourceState

# Build the state expected by content-core

state: ProcessSourceState = {
    "url": "https://example.com/article.html",
    "output_format": "markdown",   # could also be "text"

    # optional audio handling – will be filled in by Open Notebook if a model is set

}

# Perform the extraction (async)

processed = await extract_content(state)

print("Title:", processed.title)
print("Extracted markdown:", processed.content[:500])

```

### Graph Node Implementation

The internal graph node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) wraps the extraction call:

```python
async def content_process(state: SourceState) -> dict:
    # Configure content_state and add speech-to-text model

    processed_state = await extract_content(state["content_state"])
    # Error-sentinel handling omitted for brevity

    return {"content_state": processed_state}

```

### Service Layer Invocation

Create sources through the public API service for production use:

```python
from open_notebook.services import SourcesService

svc = SourcesService()
result = svc.create_source(
    notebooks=["notebook:123"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
    async_processing=True,
)

# Returns a SourceProcessingResult containing the command_id to poll later

print("Job submitted:", result.command_id)

```

## Summary

- The extraction pipeline centers on [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), which implements a LangGraph state machine coordinating the content extraction workflow.
- **content-core** provides the `extract_content()` function and `ProcessSourceState` interface, handling web pages, PDFs, Office files, images, audio, and video through unified async calls.
- Error detection relies on sentinel values (`title="Error"`) to trigger retry logic rather than persisting failures.
- The pipeline integrates with `ModelManager` to resolve speech-to-text models for audio transcription.
- Results persist through the Sources Service layer, with optional post-processing transformations available before final storage.

## Frequently Asked Questions

### What is the content-core library in Open Notebook?

The **content-core** library is a dependency (version 1.14.1) that provides the `extract_content()` function and `ProcessSourceState` type used by Open Notebook's extraction pipeline. It abstracts file-type-specific logic, allowing the application to extract clean markdown from web pages, documents, and multimedia files through a single async interface.

### How does Open Notebook handle extraction errors?

The pipeline detects soft failures by checking if the `content_core` response contains `title="Error"` and content starting with `"Failed to extract content:"`. When this sentinel is detected in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) (lines 80-90), the graph raises a `ValueError` to mark the source as retryable rather than storing the error message as a valid document.

### Can the pipeline process YouTube videos?

Yes. When processing YouTube URLs, the `content-core` library either extracts available subtitles or, when a default speech-to-text model is configured through `ModelManager`, downloads the audio stream and runs transcription via the configured provider.

### Where is the extraction graph invoked in the codebase?

The compiled `source_graph` is invoked by the **Sources Service**, typically defined in [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py). This service creates the `ProcessSourceState`, kicks off the LangGraph workflow, and returns either the completed `Source` object or a `SourceProcessingResult` containing a command ID for async job polling.