# Generating and Retrieving Source Insights with LangGraph in Open Notebook

> Learn to generate and retrieve source insights with LangGraph in Open Notebook. This guide shows how to build an LLM pipeline for content extraction, processing, and storage in SurrealDB.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: deep-dive
- Published: 2026-07-02

---

**Open Notebook leverages LangGraph's StateGraph to orchestrate an asynchronous pipeline that extracts content from URLs or files, processes them through LLM-driven transformations, and persists the results as searchable insights in SurrealDB.**

Open Notebook is an open-source knowledge management system that implements complex AI workflows using LangGraph. The platform transforms raw content into structured, AI-generated insights through a modular graph architecture defined in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py).

## Overview of the Source Processing Graph

The primary workflow for ingesting content lives in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py). This module defines a **StateGraph** that coordinates extraction, persistence, and optional transformation of source materials.

### State Management with SourceState

The graph operates on a `SourceState` typed dictionary that tracks the workflow context. This state object carries the extraction result, a list of transformations to apply, the source identifier, and a boolean flag indicating whether the content should be vector-embedded for semantic search.

### Graph Assembly and Node Functions

The source graph is assembled between lines 84-100 of [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) and follows this execution flow:

- **`content_process`** – The entry node calls **content-core** (`extract_content`) to pull text from PDFs, YouTube transcripts, or web pages, injecting model defaults for speech-to-text processing.
- **`save_source`** – Persists the `Source` record, updates its `Asset`, title, and full-text content. If `embed=True`, this node creates a vector embedding for semantic search capabilities.
- **`trigger_transformations`** – A conditional edge that checks if the user requested transformations. When transformations exist, the graph emits a `Send` for each transformation descriptor.
- **`transform_content`** – Handles the dispatched transformations by invoking the secondary transformation graph.

The compiled graph is exported as `source_graph` and invoked asynchronously via `source_graph.ainvoke(state, config)` from the API layer, ensuring the FastAPI server remains responsive during processing.

## The Transformation Graph: Creating Insights

Insights are generated by the secondary graph defined in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py). This graph receives either the full source body or a subset of text along with a `Transformation` descriptor containing a prompt template.

### LLM Integration and Response Processing

The transformation pipeline (lines 23-70) executes four critical operations:

1. **Prompt preparation** – Renders the template with input text and wraps it into a system message.
2. **LLM invocation** – `provision_langchain_model` creates a LangChain chain bound to the model selected in user settings.
3. **Response cleaning** – `clean_thinking_content` strips "thinking" artifacts from the raw output, while `extract_text_content` removes remaining markup.
4. **Insight persistence** – `await source.add_insight(transformation.title, cleaned_content)` stores the result as a `SourceInsight` record linked to the original source.

The transformation graph is compiled to `graph` and invoked by the source graph's `transform_content` node using `await transform_graph.ainvoke(...)`.

## Retrieving and Managing Insights via REST API

Once persisted, insights are exposed through the **Insights API** defined in [`api/routers/insights.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/insights.py). The router provides three primary endpoints:

- **`GET /insights/{insight_id}`** – Returns a `SourceInsightResponse` containing the insight's ID, parent source ID, type, content, and timestamps (lines 11-29).
- **`DELETE /insights/{insight_id}`** – Removes the insight from SurrealDB.
- **`POST /insights/{insight_id}/save-as-note`** – Converts an insight into a notebook `Note` for later reference.

All handlers utilize the domain model `SourceInsight` from [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py) and implement error handling through FastAPI's `HTTPException` and the custom `InvalidInputError` exception.

## End-to-End Workflow Integration

The complete pipeline integrates these components into a seamless asynchronous flow:

1. **User action** – The frontend calls the *Create Source* endpoint ([`api/routers/sources.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/sources.py)) with a URL or file and optional transformation IDs.
2. **API layer** – The endpoint creates a `Source` record, builds the initial `SourceState`, and invokes `source_graph.ainvoke`.
3. **Source graph** – Extracts content via `content_process`, persists via `save_source`, and conditionally triggers transformations.
4. **Transformation graph** – Processes text through the LLM pipeline and creates `SourceInsight` records.
5. **Persistence** – Both `Source` and `SourceInsight` objects are stored in SurrealDB; embeddings are generated if requested.
6. **Retrieval** – Clients fetch insights via the Insights router or promote them to notes using the save-as-note endpoint.

## Practical Implementation Example

The following Python snippet demonstrates how to trigger insight generation and retrieve results using the Open Notebook API:

```python
import httpx
import time

# 1️⃣ Create a source and request a transformation (e.g., "Summarize")

payload = {
    "url": "https://example.com/article.pdf",
    "apply_transformations": ["summarize"],   # IDs of Transformation records

    "embed": True,
}
resp = httpx.post("http://localhost:5055/sources", json=payload)
source_id = resp.json()["id"]

# 2️⃣ Poll until the transformation graph completes

while True:
    src = httpx.get(f"http://localhost:5055/sources/{source_id}").json()
    if src.get("insights"):
        insight_id = src["insights"][0]["id"]
        break
    time.sleep(1)

# 3️⃣ Retrieve the generated insight

insight = httpx.get(f"http://localhost:5055/insights/{insight_id}").json()
print("🔎 Insight:", insight["content"])

# 4️⃣ Convert the insight to a persistent note

note_payload = {"notebook_id": "my-notebook"}
note = httpx.post(
    f"http://localhost:5055/insights/{insight_id}/save-as-note",
    json=note_payload,
).json()
print("📝 Note created:", note["id"])

```

This workflow mirrors the Open Notebook UI behavior when users click "Generate Insight" on newly added sources. The `POST /sources` call initiates `source_graph`, while polling checks for the `insights` list populated by the transformation graph's `add_insight` method.

## Summary

- **LangGraph orchestration** – Open Notebook uses `StateGraph` in [`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py) to manage the complete content ingestion pipeline asynchronously.
- **Modular transformation** – The secondary graph in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) handles LLM interactions, response cleaning, and insight creation.
- **SurrealDB persistence** – Both sources and insights are stored as domain models (`Source`, `SourceInsight`) with optional vector embeddings for search.
- **REST API exposure** – The Insights router ([`api/routers/insights.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/insights.py)) provides endpoints for fetching, deleting, and converting insights to notes.
- **Non-blocking execution** – All graph invocations use `ainvoke()` methods to maintain FastAPI server responsiveness during long-running extractions or LLM calls.

## Frequently Asked Questions

### How does Open Notebook handle asynchronous processing of large documents?

The source graph uses `source_graph.ainvoke()` rather than synchronous invoke methods, allowing the FastAPI server to process other requests while **content-core** extracts text from large PDFs or while the LLM generates insights. The state machine pattern ensures each node (extraction, saving, transformation) completes before triggering the next, maintaining data consistency without blocking the event loop.

### What is the difference between the source graph and the transformation graph?

The **source graph** ([`open_notebook/graphs/source.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/source.py)) manages the high-level workflow: content extraction, source persistence, and orchestration. The **transformation graph** ([`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py)) is a specialized subgraph that handles LLM-specific tasks like prompt rendering via `provision_langchain_model` and response cleaning via `clean_thinking_content`. The source graph conditionally triggers the transformation graph using LangGraph's `Send` mechanism when users request specific insights.

### Can I customize the prompt templates used for insight generation?

Yes. The transformation graph reads `Transformation` descriptors that contain customizable prompt templates. These templates are rendered with the source text and wrapped into system messages before being sent to the LLM. You can define new transformations with custom prompts through the domain model, and the graph will automatically route them through `provision_langchain_model` using the user's selected model configuration.

### How are insights stored and linked to their parent sources?

Insights are persisted as `SourceInsight` records in SurrealDB through the `add_insight` method defined in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py). Each insight maintains a foreign key relationship to its parent `Source` record. When retrieved via `GET /insights/{insight_id}`, the API returns the parent source ID along with the insight content, enabling the frontend to display contextual relationships between sources and their generated insights.