# How to Process Content from Files and URLs with content-core in Open Notebook

> Learn how Open Notebook's content-core library processes files and URLs, converting raw data into searchable markdown text with a LangGraph-based pipeline.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-23

---

**Open Notebook delegates all content extraction to the `content-core` library, using a LangGraph-based pipeline in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) to convert raw files, URLs, and videos into searchable markdown text.**

Open Notebook treats every incoming document—whether a local PDF, a remote web page, or a YouTube video—as *raw content* that must be transformed into embeddable text. The heavy lifting for this conversion is performed by the **`content-core`** library, which provides a uniform `extract_content()` API across dozens of media types. The orchestration happens inside a state machine defined in the source graph, which handles parsing, error detection, and optional speech-to-text transcription before persisting the results.

## Where Content Extraction Happens

The entry point for ingestion lives in **[`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)**. The graph is built with **LangGraph** and consists of three sequential nodes:

1. **`content_process`** – prepares the state and calls `content_core.extract_content()`.
2. **`save_source`** – persists the extracted title, text, and metadata to the `Source` domain model.
3. **`transform_content`** – (optional) executes user-selected transformations on the text.

The `content_process` node is where the actual extraction occurs. It constructs a `ProcessSourceState` dictionary—containing the file path or URL plus configuration flags—and passes it to `extract_content()`:

```python

# open_notebook/graphs/source.py

from content_core import extract_content
from content_core.common import ProcessSourceState

async def content_process(state: SourceState) -> dict:
    content_state = state["content_state"]
    content_state["url_engine"] = "auto"
    content_state["document_engine"] = "auto"
    content_state["output_format"] = "markdown"

    # Optional: inject speech-to-text configuration

    try:
        model_manager = ModelManager()
        defaults = await model_manager.get_defaults()
        if defaults.default_speech_to_text_model:
            stt_model = await Model.get(defaults.default_speech_to_text_model)
            content_state["audio_provider"] = stt_model.provider
            content_state["audio_model"] = stt_model.name
    except Exception:
        pass

    # The actual extraction call

    processed_state = await extract_content(content_state)

    # Validate results

    if processed_state.title == "Error" and processed_state.content.startswith(
        "Failed to extract content:"
    ):
        raise ValueError("Could not extract content from this source")
    
    if not processed_state.content or not processed_state.content.strip():
        raise ValueError("Could not extract any text content from this source")

    return {"content_state": processed_state}

```

*The `await extract_content(content_state)` line is the sole integration point with the library.* Everything downstream—vector embedding, error handling, and transformation—operates on the `processed_state` object returned by `content_core`.

## How the Data Flows

### API Request to Graph Execution

Ingestion begins when the FastAPI endpoint in **[`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py)** receives a payload. The request is forwarded to `SourcesService`, which enqueues the source graph. The compiled workflow (`source_graph = workflow.compile()` defined around lines 184–200) receives an initial `ProcessSourceState` structure that mirrors the shape expected by `content_core`.

### The extract_content() Call

Inside `content_process`, the `content-core` library automatically selects an extraction backend based on the input type:

- **Document engine** – for PDFs, Word documents, and other local files.
- **URL engine** – for web pages and remote resources.
- **Audio/Video handler** – for YouTube URLs, which can pull existing subtitles or stream audio to a configured STT provider.

The library returns a `ProcessedState` dataclass containing:
- **`title`** – the extracted document title or `"Error"` on soft failure.
- **`content`** – the plain-text body (Markdown by default).
- **`url`** / **`file_path`** – source identifiers for provenance.

### Persistence and Transformation

The **`save_source`** node writes the cleaned text into the `Source` record ([`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py)). If the request included `embed=True`, this node triggers `source.vectorize()` to create asynchronous vector embeddings. Finally, if the user requested any transformations (e.g., summarization), the **`transform_content`** node spawns the separate `transformation` graph to process the text.

## Error Handling and Soft Failures

`content-core` never raises an exception for a missing file or unreachable URL. Instead, it returns a sentinel `ProcessedState` with `title == "Error"` and a descriptive `content` prefix. Open Notebook detects this pattern (lines 80–88 in [`source.py`](https://github.com/lfnovo/open-notebook/blob/main/source.py)) and raises a `ValueError`, marking the job as **failed** and the source as **retryable**. This prevents incomplete or corrupted data from entering the notebook.

## Configuring Extraction Engines

Default behavior is controlled by **`ContentSettings`** in [`open_notebook/domain/content_settings.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/content_settings.py). These settings define fallback values for the document and URL engines used by `content_process`. Users can override selections—such as forcing `pdfminer` for PDF processing instead of the auto-selected engine—via the Settings UI, which persists values to the database model.

## Speech-to-Text Integration

For video content lacking subtitles, Open Notebook can stream audio to an external STT provider. When a user configures a default speech-to-text model in the Settings panel, the source graph injects `audio_provider` and `audio_model` keys into the `content_state` dictionary (lines 62–71). The `content-core` library then hands these parameters to its internal audio extractor, enabling transcript generation during the extraction phase.

## Code Examples

### Directly Invoking the Source Graph

```python
from open_notebook.graphs.source import source_graph
from content_core.common import ProcessSourceState

initial_state = {
    "content_state": ProcessSourceState(
        url="https://example.com/article.html",
        # file_path="/tmp/document.pdf"  # alternative to URL

    ),
    "apply_transformations": [],
    "source_id": "source:tmp-123",
    "notebook_ids": ["notebook:default"],
    "embed": True,
}

result = await source_graph.ainvoke(initial_state)
print(result["source"].full_text[:200])

```

### Using the High-Level Service

```python
from open_notebook.services import sources_service

new_source = sources_service.create_source(
    notebooks=["notebook:default"],
    url="https://arxiv.org/pdf/2301.01234.pdf",
    embed=True,
)

print(f"Created source {new_source.id} with title '{new_source.title}'")

```

### Configuring Speech-to-Text for Video Processing

```python
from open_notebook.ai.models import Model, ModelManager

model_manager = ModelManager()
defaults = await model_manager.get_defaults()

if defaults.default_speech_to_text_model:
    stt = await Model.get(defaults.default_speech_to_text_model)
    # source.py automatically uses these for audio extraction

    print(f"Using {stt.provider}/{stt.name} for transcription")

```

## Summary

- **Single entry point**: The `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) is the only place where `content_core.extract_content()` is invoked.
- **Automatic engine selection**: `content-core` chooses the appropriate parser for documents, URLs, or video based on the input type and configuration.
- **Defensive error handling**: Soft failures return `title="Error"`, which Open Notebook converts to a `ValueError` to prevent storage of invalid data.
- **Extensible via settings**: Default engines and STT models are configurable through `ContentSettings` and injected into the extraction state.
- **LangGraph orchestration**: The entire pipeline—from download to embedding—is managed as a compiled LangGraph workflow.

## Frequently Asked Questions

### What file types does content-core support in Open Notebook?

The `content-core` library supports a wide variety of formats including PDFs, Word documents, plain text, HTML pages, and YouTube videos. According to the source graph implementation, it automatically selects a document engine for local files and a URL engine for web resources, returning everything as Markdown via the `output_format` parameter.

### How does Open Notebook handle failed extractions?

Rather than crashing, `content-core` returns a `ProcessedState` with `title == "Error"` and a message starting with `"Failed to extract content:"`. The `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) detects this pattern and raises a `ValueError`, which marks the job as failed in the queue and allows the user to retry without corrupting the notebook.

### Can I use a custom speech-to-text model for video processing?

Yes. If you configure a default speech-to-text model in the Settings UI, the source graph automatically injects `audio_provider` and `audio_model` into the `content_state` dictionary before calling `extract_content()`. This enables `content-core` to transcribe YouTube videos that lack subtitles using your chosen provider (e.g., OpenAI Whisper).

### Where are the default extraction engines configured?

Default engine selections—such as `default_content_processing_engine_doc`—are stored in the **`ContentSettings`** domain model located at [`open_notebook/domain/content_settings.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/content_settings.py). These values are read by the `content_process` node and merged into the `ProcessSourceState` before extraction begins.