Transformation Workflow Architecture in Open Notebook: A Two-Stage LangGraph Pipeline

Open Notebook processes source content through a two-stage LangGraph workflow that separates content extraction from LLM-driven transformation, storing results as insights on the source record.

The lfnovo/open-notebook repository implements a sophisticated transformation workflow architecture for processing source content using LangGraph state machines. This architecture cleanly separates concerns between content ingestion and intelligent transformation, enabling asynchronous processing of URLs, documents, and text through configurable LLM pipelines. Understanding this two-stage system is essential for extending the platform or debugging transformation pipelines.

Source Graph: Orchestrating Extraction and Transformation

The Source Graph (open_notebook/graphs/source.py) manages the initial ingestion phase. It extracts raw content, persists it to the database, and determines whether transformation pipelines should execute.

Content Processing Node

The workflow begins with the content_process node. This function calls content_core.extract_content using a configured ContentSettings object to fetch and parse source material. The node produces a ProcessSourceState containing extracted fields including content, title, and url.

# Located in open_notebook/graphs/source.py

# The content_process node populates the state with extracted metadata

state = {
    "content_state": {
        "url": "https://example.com/article",
        "output_format": "markdown",
    },
    # ... other state keys

}

Source Persistence and Embedding

Next, the save_source node persists the extracted text to a Source record and updates its associated Asset. If vector embedding is requested (via the embed state flag), the node invokes source.vectorize() to generate and store embeddings for semantic search.

Conditional Transformation Trigger

The graph uses conditional edges to determine transformation execution. The trigger_transformations function (lines 44-59) inspects the apply_transformations state field. When transformations are requested, it builds a list of Send messages—one per transformation—and routes to the transform_content node.

The edge configuration (lines 92-100) defines this flow:

workflow.add_edge(START, "content_process")
workflow.add_edge("content_process", "save_source")
workflow.add_conditional_edges(
    "save_source", trigger_transformations, ["transform_content"]
)
workflow.add_edge("transform_content", END)

The transform_content node forwards the source and specific Transformation record to the Transformation Graph for execution.

Transformation Graph: Executing LLM-Driven Transformations

The Transformation Graph (open_notebook/graphs/transformation.py, lines 1-76) handles the actual LLM processing. Unlike the Source Graph's multi-node flow, this graph uses a single node compiled via graph = agent_state.compile() (line 75).

The Run Transformation Node

The run_transformation node executes four distinct operations:

1. Prompt Construction: Builds a system prompt combining the Transformation.prompt template with optional DefaultPrompts.transformation_instructions.
2. Model Provisioning: Calls provision_langchain_model to instantiate a LangChain chain configured for the "transformation" model context.
3. LLM Invocation: Sends a SystemMessage containing the prompt and a HumanMessage containing the source text.
4. Output Processing: Extracts clean text using extract_text_content and clean_thinking_content utilities, then stores the result via source.add_insight.

Prompt Construction and Model Provisioning

The transformation leverages the Transformation domain model (open_notebook/domain/transformation.py) which defines the Pydantic schema for transformation templates. The system supports optional default instructions through DefaultPrompts, allowing users to override or extend base behavior per transformation instance.

End-to-End Execution Flow

The complete transformation workflow architecture operates as follows:

1. Trigger: User or API calls the source pipeline (e.g., via api/sources_service.py).
2. Extraction: The Source Graph executes content_process → save_source to extract and persist content.
3. Decision: The trigger_transformations conditional edge checks for attached Transformation objects.
4. Execution: For each transformation, the Source Graph sends a message to the Transformation Graph, which runs run_transformation to invoke the LLM.
5. Storage: The Transformation Graph writes the cleaned LLM response back to the source as an insight using source.add_insight.

This design ensures that extraction and embedding occur independently of LLM processing, allowing the system to scale ingestion and transformation separately.

Implementation Details and Code Structure

Key files comprising the transformation workflow architecture:

• open_notebook/graphs/source.py: Main orchestration graph with nodes for content_process, save_source, and transform_content.
• open_notebook/graphs/transformation.py: Single-node graph (run_transformation) compiled at line 75.
• open_notebook/domain/transformation.py: Domain model defining transformation schemas and default prompts.
• open_notebook/ai/provision.py: Supplies the LangChain model configuration for transformation steps.
• open_notebook/utils/text_utils.py: Provides extract_text_content and clean_thinking_content for sanitizing LLM outputs.
• api/sources_service.py and api/transformations_service.py: API layer entry points.

Example invocation of the complete pipeline:

import asyncio
from open_notebook.graphs.source import source_graph

async def ingest(url: str):
    state = {
        "content_state": {
            "url": url,
            "output_format": "markdown",
        },
        "apply_transformations": [],  # or list of Transformation objects

        "source_id": "new",
        "notebook_ids": ["notebook-1"],
        "embed": True,
        "source": None,
        "transformation": [],
    }
    result = await source_graph.ainvoke(state)
    return result

# Run the pipeline

asyncio.run(ingest("https://example.com/article"))

Summary

• Open Notebook uses a two-stage LangGraph architecture separating source ingestion (open_notebook/graphs/source.py) from LLM transformation (open_notebook/graphs/transformation.py).
• The Source Graph handles extraction, persistence, and conditional routing via content_process, save_source, and trigger_transformations nodes.
• The Transformation Graph executes single transformations through the run_transformation node, which provisions models, constructs prompts, and stores insights.
• State management flows asynchronously through ProcessSourceState, allowing parallel processing of multiple transformations per source.
• Results persist as insights on the source record using source.add_insight, creating a searchable knowledge base.

Frequently Asked Questions

What is the difference between the Source Graph and Transformation Graph?

The Source Graph (open_notebook/graphs/source.py) orchestrates the entire ingestion pipeline including content extraction, database persistence, and vector embedding. It decides whether transformations should run. The Transformation Graph (open_notebook/graphs/transformation.py) is a specialized, single-node graph that only executes LLM prompts against extracted text. This separation allows the Source Graph to handle multiple transformations sequentially while maintaining clean separation between extraction logic and AI processing.

How does Open Notebook handle multiple transformations on a single source?

The trigger_transformations function (lines 44-59) in source.py generates a list of Send messages—one for each transformation specified in the apply_transformations state field. The Source Graph invokes the transform_content node for each message, which calls the Transformation Graph individually. This architecture allows different transformations to run independently, with each output stored as a separate insight on the same source record.

What happens to the output of a transformation?

The Transformation Graph processes the LLM response through extract_text_content and clean_thinking_content utilities to remove artifacts and formatting. The cleaned text is then attached to the source via source.add_insight, creating a persistent insight record associated with the original source material. These insights become available for retrieval in notebooks and search interfaces.

How is the LLM model selected for transformations?

The run_transformation node calls provision_langchain_model from open_notebook/ai/provision.py, specifically requesting the model configuration designated for "transformation" contexts. This abstraction allows the system to use different models or providers for transformations versus other operations (like chat or embedding), configured through the application's AI provisioning settings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}