# How LangGraph Orchestrates Chat, Source Ingestion, and Transformations in Open‑Notebook

> Discover how LangGraph orchestrates chat, source ingestion, and transformations in open-notebook. Learn about state machines managing source lifecycle and insight generation for conversational retrieval.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-06-22

---

**Open‑Notebook leverages LangGraph to coordinate three interconnected workflows—content extraction, vector embedding, and LLM‑powered transformations—through two distinct state machines that manage source lifecycle and insight generation for conversational retrieval.**

The open‑notebook repository uses LangGraph to model complex asynchronous operations as stateful graphs. This architecture ensures reliable ingestion of external sources, optional embedding for semantic search, and user‑defined text transformations that feed directly into chat contexts.

## The Dual‑Graph Architecture

Instead of monolithic pipelines, open‑notebook implements two specialized graphs in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py). The `source_graph` manages the ingestion lifecycle, while the `transform_graph` handles LLM‑based enrichment. Together they bridge raw content acquisition and conversational AI.

## Source Ingestion Workflow

The `source_graph` orchestrates the first phase of the pipeline, processing URLs, PDFs, or videos into storable, searchable, and transformable assets.

### Content Extraction Node

The `content_process` node initializes a `ProcessSourceState` with default processing engines and any configured Speech‑to‑Text model from the `ModelManager`. It invokes `extract_content` to fetch and normalize raw text using the **content‑core** library. Errors are captured and converted to `ValueError` instances, enabling LangGraph’s built‑in retry mechanisms.

```python

# Conceptual state initialization for content extraction

state = ProcessSourceState(
    source_id="src_456",
    url="https://example.com/article",
    processing_engines=["html", "pdf"],
    stt_model=None
)

# content_process returns updated state with extracted text

```

### Persistence and Vectorization

Following extraction, the `save_source` node loads the `Source` record by `source_id`, updates its `asset`, `full_text`, and title fields, and conditionally triggers embedding generation. When `embed=True`, the node calls `source.vectorize()` to create vector representations for retrieval‑augmented generation (RAG) within chat contexts.

```python

# Inside save_source node implementation

source = Source.get_by_id(state["source_id"])
source.full_text = extracted_content
source.asset = asset_metadata
if state.get("embed"):
    source.vectorize()  # Async embedding job

source.save()

```

### Transformation Dispatch

The `trigger_transformations` node dynamically spawns parallel transformation tasks using LangGraph’s `Send` API. For each `Transformation` supplied in the input, it creates a dispatch command that routes to the `transform_content` node, which forwards execution to the `transform_graph`.

```python
from langgraph.constants import Send

def trigger_transformations(state: ProcessSourceState):
    tasks = [
        Send("transform_content", {"transformation": t, "source_id": state["source_id"]})
        for t in state["transformations"]
    ]
    return tasks

```

## Transformation Execution Pipeline

The `transform_graph` defined in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) executes isolated LLM calls that convert raw text into structured insights.

### LLM Insight Generation

The `run_transformation` node renders the transformation’s Jinja2 template using current state variables (`input_text`, `source`, etc.), then invokes `provision_langchain_model` to instantiate the correct provider through the **Esperanto** AI abstraction layer. The resulting LangChain runnable chain processes the prompt, and the cleaned response is stored as a `SourceInsight`.

```python

# Transformation execution flow

template = transformation.template
system_prompt = template.render(
    input_text=source.full_text,
    source_title=source.title
)

# Provision model via Esperanto abstraction

llm = provision_langchain_model(transformation.model_config)
response = llm.invoke(system_prompt)

# Store insight

SourceInsight.create(
    source_id=source.id,
    content=response.content,
    transformation_id=transformation.id
)

```

## Chat Context Integration

Once persisted, sources and their associated insights become accessible to the chat interface. The `source_graph` ensures that ingested content is immediately available for retrieval, while `transform_graph` outputs enrich the conversation with pre‑computed analyses. Vectorized sources enable semantic search within chat sessions, and `SourceInsight` objects provide pre‑generated summaries that reduce latency during active conversations.

## Summary

- **Dual‑graph design**: Open‑notebook separates ingestion (`source_graph`) from enrichment (`transform_graph`) to isolate failure domains and enable independent scaling.
- **Stateful extraction**: The `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) uses `ProcessSourceState` to manage content‑core extraction and Speech‑to‑Text processing.
- **Async embedding**: The `save_source` node triggers `source.vectorize()` for semantic search capabilities inside chat contexts.
- **Dynamic dispatch**: LangGraph’s `Send` API enables parallel execution of multiple transformations via `trigger_transformations`.
- **Model abstraction**: `provision_langchain_model` leverages the Esperanto framework to support multiple LLM providers without graph modifications.

## Frequently Asked Questions

### How does LangGraph handle failures during content extraction?

The `content_process` node in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) catches extraction errors and raises them as `ValueError` exceptions. This allows LangGraph’s built‑in retry policies to re‑execute the node with exponential backoff, ensuring transient network failures during URL fetching or PDF processing do not terminate the entire workflow.

### What is the relationship between source ingestion and chat context?

Sources processed by the `source_graph` are immediately persisted to the database and optionally vectorized. The chat interface queries these records through the `Source` model, enabling retrieval‑augmented generation. Additionally, `SourceInsight` objects generated by the `transform_graph` provide pre‑computed context that the chat system injects into prompts to ground responses.

### Can multiple transformations run simultaneously on a single source?

Yes. The `trigger_transformations` node uses LangGraph’s `Send` primitive to create parallel task dispatches for each `Transformation` in the input list. Each dispatch executes the `transform_content` node independently, allowing multiple LLM analyses (summarization, keyword extraction, translation) to run concurrently against the same source text.

### How does the system determine which LLM to use for transformations?

The `run_transformation` node calls `provision_langchain_model`, which interfaces with the **Esperanto** AI abstraction layer. This function selects the appropriate provider and model based on the `ModelManager` configuration and the specific `Transformation` requirements, returning a LangChain runnable chain that standardizes invocation across OpenAI, Anthropic, or local model deployments.