# Transformation Workflow Architecture in Open Notebook: A Two-Stage LangGraph Pipeline

> Explore Open Notebook's transformation workflow architecture. This two-stage LangGraph pipeline efficiently extracts and transforms source content into valuable insights.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: architecture
- Published: 2026-06-19

---

**Open Notebook processes source content through a two-stage LangGraph workflow that separates content extraction from LLM-driven transformation, storing results as insights on the source record.**

The `lfnovo/open-notebook` repository implements a sophisticated transformation workflow architecture for processing source content using LangGraph state machines. This architecture cleanly separates concerns between content ingestion and intelligent transformation, enabling asynchronous processing of URLs, documents, and text through configurable LLM pipelines. Understanding this two-stage system is essential for extending the platform or debugging transformation pipelines.

## Source Graph: Orchestrating Extraction and Transformation

The **Source Graph** ([`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)) manages the initial ingestion phase. It extracts raw content, persists it to the database, and determines whether transformation pipelines should execute.

### Content Processing Node

The workflow begins with the `content_process` node. This function calls `content_core.extract_content` using a configured `ContentSettings` object to fetch and parse source material. The node produces a `ProcessSourceState` containing extracted fields including `content`, `title`, and `url`.

```python

# Located in open_notebook/graphs/source.py

# The content_process node populates the state with extracted metadata

state = {
    "content_state": {
        "url": "https://example.com/article",
        "output_format": "markdown",
    },
    # ... other state keys

}

```

### Source Persistence and Embedding

Next, the `save_source` node persists the extracted text to a `Source` record and updates its associated `Asset`. If vector embedding is requested (via the `embed` state flag), the node invokes `source.vectorize()` to generate and store embeddings for semantic search.

### Conditional Transformation Trigger

The graph uses conditional edges to determine transformation execution. The `trigger_transformations` function (lines 44-59) inspects the `apply_transformations` state field. When transformations are requested, it builds a list of `Send` messages—one per transformation—and routes to the `transform_content` node.

The edge configuration (lines 92-100) defines this flow:

```python
workflow.add_edge(START, "content_process")
workflow.add_edge("content_process", "save_source")
workflow.add_conditional_edges(
    "save_source", trigger_transformations, ["transform_content"]
)
workflow.add_edge("transform_content", END)

```

The `transform_content` node forwards the source and specific `Transformation` record to the **Transformation Graph** for execution.

## Transformation Graph: Executing LLM-Driven Transformations

The **Transformation Graph** ([`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py), lines 1-76) handles the actual LLM processing. Unlike the Source Graph's multi-node flow, this graph uses a single node compiled via `graph = agent_state.compile()` (line 75).

### The Run Transformation Node

The `run_transformation` node executes four distinct operations:

1. **Prompt Construction**: Builds a system prompt combining the `Transformation.prompt` template with optional `DefaultPrompts.transformation_instructions`.
2. **Model Provisioning**: Calls `provision_langchain_model` to instantiate a LangChain chain configured for the "transformation" model context.
3. **LLM Invocation**: Sends a `SystemMessage` containing the prompt and a `HumanMessage` containing the source text.
4. **Output Processing**: Extracts clean text using `extract_text_content` and `clean_thinking_content` utilities, then stores the result via `source.add_insight`.

### Prompt Construction and Model Provisioning

The transformation leverages the `Transformation` domain model ([`open_notebook/domain/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/transformation.py)) which defines the Pydantic schema for transformation templates. The system supports optional default instructions through `DefaultPrompts`, allowing users to override or extend base behavior per transformation instance.

## End-to-End Execution Flow

The complete transformation workflow architecture operates as follows:

1. **Trigger**: User or API calls the source pipeline (e.g., via [`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py)).
2. **Extraction**: The **Source Graph** executes `content_process` → `save_source` to extract and persist content.
3. **Decision**: The `trigger_transformations` conditional edge checks for attached `Transformation` objects.
4. **Execution**: For each transformation, the **Source Graph** sends a message to the **Transformation Graph**, which runs `run_transformation` to invoke the LLM.
5. **Storage**: The Transformation Graph writes the cleaned LLM response back to the source as an insight using `source.add_insight`.

This design ensures that extraction and embedding occur independently of LLM processing, allowing the system to scale ingestion and transformation separately.

## Implementation Details and Code Structure

Key files comprising the transformation workflow architecture:

- **[`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)**: Main orchestration graph with nodes for `content_process`, `save_source`, and `transform_content`.
- **[`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py)**: Single-node graph (`run_transformation`) compiled at line 75.
- **[`open_notebook/domain/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/transformation.py)**: Domain model defining transformation schemas and default prompts.
- **[`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py)**: Supplies the LangChain model configuration for transformation steps.
- **[`open_notebook/utils/text_utils.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/utils/text_utils.py)**: Provides `extract_text_content` and `clean_thinking_content` for sanitizing LLM outputs.
- **[`api/sources_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/sources_service.py)** and **[`api/transformations_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/transformations_service.py)**: API layer entry points.

Example invocation of the complete pipeline:

```python
import asyncio
from open_notebook.graphs.source import source_graph

async def ingest(url: str):
    state = {
        "content_state": {
            "url": url,
            "output_format": "markdown",
        },
        "apply_transformations": [],  # or list of Transformation objects

        "source_id": "new",
        "notebook_ids": ["notebook-1"],
        "embed": True,
        "source": None,
        "transformation": [],
    }
    result = await source_graph.ainvoke(state)
    return result

# Run the pipeline

asyncio.run(ingest("https://example.com/article"))

```

## Summary

- **Open Notebook uses a two-stage LangGraph architecture** separating source ingestion ([`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)) from LLM transformation ([`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py)).
- **The Source Graph handles extraction, persistence, and conditional routing** via `content_process`, `save_source`, and `trigger_transformations` nodes.
- **The Transformation Graph executes single transformations** through the `run_transformation` node, which provisions models, constructs prompts, and stores insights.
- **State management flows asynchronously** through `ProcessSourceState`, allowing parallel processing of multiple transformations per source.
- **Results persist as insights** on the source record using `source.add_insight`, creating a searchable knowledge base.

## Frequently Asked Questions

### What is the difference between the Source Graph and Transformation Graph?

The **Source Graph** ([`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)) orchestrates the entire ingestion pipeline including content extraction, database persistence, and vector embedding. It decides whether transformations should run. The **Transformation Graph** ([`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py)) is a specialized, single-node graph that only executes LLM prompts against extracted text. This separation allows the Source Graph to handle multiple transformations sequentially while maintaining clean separation between extraction logic and AI processing.

### How does Open Notebook handle multiple transformations on a single source?

The `trigger_transformations` function (lines 44-59) in [`source.py`](https://github.com/lfnovo/open-notebook/blob/main/source.py) generates a list of `Send` messages—one for each transformation specified in the `apply_transformations` state field. The Source Graph invokes the `transform_content` node for each message, which calls the Transformation Graph individually. This architecture allows different transformations to run independently, with each output stored as a separate insight on the same source record.

### What happens to the output of a transformation?

The Transformation Graph processes the LLM response through `extract_text_content` and `clean_thinking_content` utilities to remove artifacts and formatting. The cleaned text is then attached to the source via `source.add_insight`, creating a persistent insight record associated with the original source material. These insights become available for retrieval in notebooks and search interfaces.

### How is the LLM model selected for transformations?

The `run_transformation` node calls `provision_langchain_model` from [`open_notebook/ai/provision.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/ai/provision.py), specifically requesting the model configuration designated for "transformation" contexts. This abstraction allows the system to use different models or providers for transformations versus other operations (like chat or embedding), configured through the application's AI provisioning settings.