# Integration Pattern Between content‑core and the Source Ingestion Pipeline in Open Notebook

> Discover the integration pattern between content-core and the source ingestion pipeline in Open Notebook. Learn how content-core acts as a black-box service within a three-step state-transformation pattern.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-19

---

**Open Notebook treats the content‑core library as a black‑box extraction service that is invoked from within its own LangGraph‑based source‑ingestion workflow, following a strict three‑step state‑transformation pattern.**

The **integration pattern between content‑core and the source ingestion pipeline** in the `lfnovo/open-notebook` repository demonstrates a clean architectural separation between content extraction and domain persistence. By treating **content‑core** as an external service consumed within a LangGraph state machine, the system maintains modularity while handling complex document processing, OCR, and transcription tasks. This design allows the ingestion workflow to remain agnostic to the underlying content formats while ensuring type‑safe data flow between components.

## Architecture Overview

Content extraction is handled as an async workflow orchestrated by `source_graph` in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py). The pipeline treats **content‑core** as a standalone extraction engine that receives a typed configuration object and returns processed content, creating a clear boundary between extraction logic and Open Notebook's domain concerns such as embedding generation and notebook linking.

## The Three‑Step Integration Pattern

The integration follows a strict sequence of state transformations that map directly to LangGraph nodes.

### Step 1: Preparing the ProcessSourceState Payload

The `content_process` node constructs a dictionary matching the `ProcessSourceState` schema expected by content‑core. Located at lines 34‑62 of [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py), this preparation includes default processing engines, output format selection, and injection of speech‑to‑text model configuration from the Open Notebook `ModelManager`.

```python
content_state = {
    "url": source_url,
    "output_format": "markdown",
    "url_engine": "auto",
    "document_engine": "auto",
    # Optional speech-to-text model injected from ModelManager

    "audio_provider": stt_model.provider,
    "audio_model": stt_model.name,
}

```

### Step 2: Invoking content_core.extract_content

The same node calls `extract_content` from the content‑core package (imported at lines 4‑5 of [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)). This function performs the heavy lifting—including URL downloads, file parsing, OCR, and audio transcription—and returns a populated `ProcessSourceState` object containing `title`, `content`, `url`, `file_path`, and error metadata.

```python
processed_state = await extract_content(content_state)

```

### Step 3: Handling Results and Persistence

The workflow transitions to the `save_source` node (lines 78‑92), which persists the extracted data to the `Source` record and triggers embedding creation. The node validates the extraction result before saving, ensuring content integrity before linking to notebooks.

```python
source.full_text = processed_state.content
source.title = processed_state.title
await source.save()

```

## Error Handling and Workflow Control

The pipeline implements strict failure detection at the persistence layer. If `extract_content` returns a title prefixed with `"Error"` or containing `"Failed to extract content:"`, the `save_source` node raises a `ValueError`, causing LangGraph to mark the job as failed and retryable. This conditional edge logic ensures that malformed extractions do not propagate into the notebook's knowledge base.

## Graph Composition and Conditional Processing

The `source_graph` wires together async nodes in sequence: `content_process → save_source → optional transform_content`. **Conditional edges** determine whether transformation nodes execute based on request parameters, allowing the pipeline to handle optional post‑processing (such as content summarization or restructuring) without complicating the core extraction flow.

## Key Implementation Files

The integration spans several critical files within the repository:

- **[`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)**: Defines the LangGraph workflow, the `content_process` node (lines 34‑62), and error handling logic (lines 78‑92).
- **[`commands/source_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/source_commands.py)**: Entry point that triggers the ingestion pipeline.
- **content‑core package**: External dependency imported at lines 4‑5 of [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) providing the `extract_content` function.

## Summary

- Open Notebook treats **content‑core** as a black‑box extraction service, invoking it through a single typed call to `extract_content`.
- The **integration pattern** follows three distinct phases: state preparation, external extraction, and result persistence.
- **LangGraph nodes** (`content_process` and `save_source`) orchestrate the workflow defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py).
- **Error detection** occurs at the persistence layer, raising `ValueError` for failed extractions to enable retry logic.
- **Conditional edges** allow optional transformations to run after successful persistence without blocking the core ingestion flow.

## Frequently Asked Questions

### How does Open Notebook handle failures during content extraction?

When the `save_source` node detects an error indicator in the returned title (specifically the string `"Failed to extract content:"`), it raises a `ValueError` that marks the LangGraph job as failed. This design makes the extraction step retryable while preventing corrupted data from entering the notebook's storage.

### What is the role of the ProcessSourceState in the integration?

The `ProcessSourceState` acts as the contract between Open Notebook and content‑core. It is a typed dictionary prepared in the `content_process` node that configures extraction engines, output formats, and audio model settings, ensuring that content‑core receives all necessary parameters to process URLs, files, or audio streams.

### Where is the source ingestion pipeline triggered in the codebase?

The pipeline is initiated from [`commands/source_commands.py`](https://github.com/lfnovo/open-notebook/blob/main/commands/source_commands.py), which constructs the initial context and invokes the `source_graph` defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py). This command layer serves as the entry point for user‑initiated or automated ingestion tasks.

### Why does the architecture separate content extraction from persistence?

Separating these concerns through the **content‑core** integration allows the extraction logic to evolve independently—supporting new file formats or OCR engines—without modifying Open Notebook's domain models. The LangGraph workflow acts as a thin orchestration layer that handles only state management, persistence, and error handling, keeping the system modular and testable.