Generating and Retrieving Source Insights with LangGraph in Open Notebook

Open Notebook leverages LangGraph's StateGraph to orchestrate an asynchronous pipeline that extracts content from URLs or files, processes them through LLM-driven transformations, and persists the results as searchable insights in SurrealDB.

Open Notebook is an open-source knowledge management system that implements complex AI workflows using LangGraph. The platform transforms raw content into structured, AI-generated insights through a modular graph architecture defined in open_notebook/graphs/source.py and open_notebook/graphs/transformation.py.

Overview of the Source Processing Graph

The primary workflow for ingesting content lives in open_notebook/graphs/source.py. This module defines a StateGraph that coordinates extraction, persistence, and optional transformation of source materials.

State Management with SourceState

The graph operates on a SourceState typed dictionary that tracks the workflow context. This state object carries the extraction result, a list of transformations to apply, the source identifier, and a boolean flag indicating whether the content should be vector-embedded for semantic search.

Graph Assembly and Node Functions

The source graph is assembled between lines 84-100 of open_notebook/graphs/source.py and follows this execution flow:

• content_process – The entry node calls content-core (extract_content) to pull text from PDFs, YouTube transcripts, or web pages, injecting model defaults for speech-to-text processing.
• save_source – Persists the Source record, updates its Asset, title, and full-text content. If embed=True, this node creates a vector embedding for semantic search capabilities.
• trigger_transformations – A conditional edge that checks if the user requested transformations. When transformations exist, the graph emits a Send for each transformation descriptor.
• transform_content – Handles the dispatched transformations by invoking the secondary transformation graph.

The compiled graph is exported as source_graph and invoked asynchronously via source_graph.ainvoke(state, config) from the API layer, ensuring the FastAPI server remains responsive during processing.

The Transformation Graph: Creating Insights

Insights are generated by the secondary graph defined in open_notebook/graphs/transformation.py. This graph receives either the full source body or a subset of text along with a Transformation descriptor containing a prompt template.

LLM Integration and Response Processing

The transformation pipeline (lines 23-70) executes four critical operations:

1. Prompt preparation – Renders the template with input text and wraps it into a system message.
2. LLM invocation – provision_langchain_model creates a LangChain chain bound to the model selected in user settings.
3. Response cleaning – clean_thinking_content strips "thinking" artifacts from the raw output, while extract_text_content removes remaining markup.
4. Insight persistence – await source.add_insight(transformation.title, cleaned_content) stores the result as a SourceInsight record linked to the original source.

The transformation graph is compiled to graph and invoked by the source graph's transform_content node using await transform_graph.ainvoke(...).

Retrieving and Managing Insights via REST API

Once persisted, insights are exposed through the Insights API defined in api/routers/insights.py. The router provides three primary endpoints:

• GET /insights/{insight_id} – Returns a SourceInsightResponse containing the insight's ID, parent source ID, type, content, and timestamps (lines 11-29).
• DELETE /insights/{insight_id} – Removes the insight from SurrealDB.
• POST /insights/{insight_id}/save-as-note – Converts an insight into a notebook Note for later reference.

All handlers utilize the domain model SourceInsight from open_notebook/domain/notebook.py and implement error handling through FastAPI's HTTPException and the custom InvalidInputError exception.

End-to-End Workflow Integration

The complete pipeline integrates these components into a seamless asynchronous flow:

1. User action – The frontend calls the Create Source endpoint (api/routers/sources.py) with a URL or file and optional transformation IDs.
2. API layer – The endpoint creates a Source record, builds the initial SourceState, and invokes source_graph.ainvoke.
3. Source graph – Extracts content via content_process, persists via save_source, and conditionally triggers transformations.
4. Transformation graph – Processes text through the LLM pipeline and creates SourceInsight records.
5. Persistence – Both Source and SourceInsight objects are stored in SurrealDB; embeddings are generated if requested.
6. Retrieval – Clients fetch insights via the Insights router or promote them to notes using the save-as-note endpoint.

Practical Implementation Example

The following Python snippet demonstrates how to trigger insight generation and retrieve results using the Open Notebook API:

import httpx
import time

# 1️⃣ Create a source and request a transformation (e.g., "Summarize")

payload = {
    "url": "https://example.com/article.pdf",
    "apply_transformations": ["summarize"],   # IDs of Transformation records

    "embed": True,
}
resp = httpx.post("http://localhost:5055/sources", json=payload)
source_id = resp.json()["id"]

# 2️⃣ Poll until the transformation graph completes

while True:
    src = httpx.get(f"http://localhost:5055/sources/{source_id}").json()
    if src.get("insights"):
        insight_id = src["insights"][0]["id"]
        break
    time.sleep(1)

# 3️⃣ Retrieve the generated insight

insight = httpx.get(f"http://localhost:5055/insights/{insight_id}").json()
print("🔎 Insight:", insight["content"])

# 4️⃣ Convert the insight to a persistent note

note_payload = {"notebook_id": "my-notebook"}
note = httpx.post(
    f"http://localhost:5055/insights/{insight_id}/save-as-note",
    json=note_payload,
).json()
print("📝 Note created:", note["id"])

This workflow mirrors the Open Notebook UI behavior when users click "Generate Insight" on newly added sources. The POST /sources call initiates source_graph, while polling checks for the insights list populated by the transformation graph's add_insight method.

Summary

• LangGraph orchestration – Open Notebook uses StateGraph in open_notebook/graphs/source.py to manage the complete content ingestion pipeline asynchronously.
• Modular transformation – The secondary graph in open_notebook/graphs/transformation.py handles LLM interactions, response cleaning, and insight creation.
• SurrealDB persistence – Both sources and insights are stored as domain models (Source, SourceInsight) with optional vector embeddings for search.
• REST API exposure – The Insights router (api/routers/insights.py) provides endpoints for fetching, deleting, and converting insights to notes.
• Non-blocking execution – All graph invocations use ainvoke() methods to maintain FastAPI server responsiveness during long-running extractions or LLM calls.

Frequently Asked Questions

How does Open Notebook handle asynchronous processing of large documents?

The source graph uses source_graph.ainvoke() rather than synchronous invoke methods, allowing the FastAPI server to process other requests while content-core extracts text from large PDFs or while the LLM generates insights. The state machine pattern ensures each node (extraction, saving, transformation) completes before triggering the next, maintaining data consistency without blocking the event loop.

What is the difference between the source graph and the transformation graph?

The source graph (open_notebook/graphs/source.py) manages the high-level workflow: content extraction, source persistence, and orchestration. The transformation graph (open_notebook/graphs/transformation.py) is a specialized subgraph that handles LLM-specific tasks like prompt rendering via provision_langchain_model and response cleaning via clean_thinking_content. The source graph conditionally triggers the transformation graph using LangGraph's Send mechanism when users request specific insights.

Can I customize the prompt templates used for insight generation?

Yes. The transformation graph reads Transformation descriptors that contain customizable prompt templates. These templates are rendered with the source text and wrapped into system messages before being sent to the LLM. You can define new transformations with custom prompts through the domain model, and the graph will automatically route them through provision_langchain_model using the user's selected model configuration.

How are insights stored and linked to their parent sources?

Insights are persisted as SourceInsight records in SurrealDB through the add_insight method defined in open_notebook/domain/notebook.py. Each insight maintains a foreign key relationship to its parent Source record. When retrieved via GET /insights/{insight_id}, the API returns the parent source ID along with the insight content, enabling the frontend to display contextual relationships between sources and their generated insights.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}