How Open Notebook Applies Transformations to Sources Using the transformation.py Graph

Open Notebook applies user-defined LLM transformations to sources by executing a compiled LangGraph state machine in open_notebook/graphs/transformation.py that resolves inputs, builds prompts, calls the LLM, and persists results as insights.

Open Notebook enables intelligent document processing through a dedicated transformation system that executes user-defined prompts against source content. The core orchestration happens in the transformation.py graph, a minimal LangGraph implementation that handles the entire lifecycle from input validation to insight persistence. This architecture leverages FastAPI's async capabilities and SurrealDB's command queue to provide a responsive, fire-and-forget transformation pipeline.

The Transformation Execution Architecture

The transformation process follows a linear pipeline that begins with an HTTP request and terminates with persisted insights. Understanding this flow is essential for debugging custom transformations or extending the system.

API Entry Point and Graph Invocation

Transformations are triggered through the REST endpoint POST /transformations/execute defined in api/routers/transformations.py. This endpoint validates the requested Transformation object and target model, then invokes the compiled graph asynchronously:

result = await transformation_graph.ainvoke(
    {"input_text": request.input_text,
     "transformation": transformation},
    config={"configurable": {"model_id": request.model_id}},
)

The transformation_graph object is the compiled LangGraph instance exported from open_notebook/graphs/transformation.py.

Graph Structure and Compilation

The transformation graph is a minimal StateGraph compiled at module import time via graph = agent_state.compile(). The topology consists of a single execution node:

START → agent → END

The agent node executes the run_transformation function, which contains the core business logic. This design ensures predictable, linear execution without conditional branching or cycles, making the transformation process straightforward to trace and debug.

The run_transformation Node Implementation

The run_transformation function in open_notebook/graphs/transformation.py performs six distinct operations to process source content:

1. Input Resolution

The node first retrieves the source object and content from the graph state. If input_text is not provided, it automatically falls back to source.full_text:

source_obj = state.get("source")
content = state.get("input_text") or source_obj.full_text

This ensures transformations can operate on either explicit text inputs or existing source documents.

2. System Prompt Construction

The node builds the final prompt by combining the transformation's template with global instructions. If DefaultPrompts.transformation_instructions exists, it prepends these instructions to the user-defined prompt, followed by the delimiter # INPUT:

system_prompt = Prompter(template_text=transformation_template_text).render(data=state)

3. LLM Invocation via Esperanto

The graph utilizes the provision_langchain_model helper to instantiate a LangChain chain configured with the model_id supplied via configurable.model_id. The payload consists of a SystemMessage containing the constructed prompt and a HumanMessage containing the source text:

chain = await provision_langchain_model(...)
response = await chain.ainvoke(payload)

4. Output Cleaning

Raw LLM responses are processed through extract_text_content and clean_thinking_content to remove artifacts, thinking tokens, and formatting noise, yielding plain text suitable for storage.

5. Insight Persistence

If a Source object was provided in the state, the cleaned output is persisted as an insight using source.add_insight(transformation.title, cleaned_content). This method queues a create_insight command in SurrealDB's async command queue, enabling fire-and-forget persistence without blocking the response:

await source.add_insight(transformation.title, cleaned_content)

6. Result Propagation

Finally, the node returns a dictionary containing the cleaned output, which propagates back to the API caller:

return {"output": cleaned_content}

Practical Implementation Examples

Calling the Transformation Endpoint

Execute a transformation via the REST API by posting to the transformations router:

import httpx

payload = {
    "input_text": "Summarize the key findings of this PDF.",
    "transformation_id": "transformation:abc123",
    "model_id": "model:openai-gpt-4o"
}
resp = httpx.post("http://localhost:5055/transformations/execute", json=payload)
print(resp.json()["output"])

This returns the LLM-generated content and, if a source was specified, stores the result as an insight.

Direct Graph Invocation

For programmatic access within the Open Notebook codebase, import and invoke the graph directly:

from open_notebook.graphs.transformation import graph as transformation_graph
from open_notebook.domain.transformation import Transformation
from open_notebook.domain.notebook import Source

state = {
    "input_text": None,  # Uses source.full_text automatically

    "source": source,
    "transformation": transformation,
}
config = {"configurable": {"model_id": "model:anthropic-claude-3"}}

result = await transformation_graph.ainvoke(state, config=config)
print("Insight generated:", result["output"])

Customizing Default Instructions

Modify global transformation behavior by updating the DefaultPrompts singleton:

from open_notebook.domain.transformation import DefaultPrompts

default = await DefaultPrompts.get_instance()
default.transformation_instructions = "Always answer in bullet points."
await default.update()

This prepends the instruction to all subsequent transformation prompts.

Summary

• Open Notebook applies transformations through a compiled LangGraph in open_notebook/graphs/transformation.py with a linear START → agent → END topology.
• The run_transformation node resolves inputs, constructs system prompts using Prompter, and invokes LLMs via provision_langchain_model.
• Results are cleaned using clean_thinking_content and persisted via source.add_insight, which queues async SurrealDB commands.
• The REST endpoint POST /transformations/execute in api/routers/transformations.py provides the primary entry point for external clients.
• Global instructions can be configured through the DefaultPrompts model to prepend standard formatting to all transformations.

Frequently Asked Questions

What is the transformation.py graph in Open Notebook?

The transformation.py graph is a minimal LangGraph StateGraph defined in open_notebook/graphs/transformation.py that orchestrates LLM-based text transformations. It consists of a single agent node executing the run_transformation function, compiled at import time to handle the complete workflow from prompt construction to insight persistence.

How does the transformation graph handle source content?

The graph accepts source content through the input_text state field or automatically resolves it from source.full_text if the field is empty. This dual-input design allows the transformation system to process either ad-hoc text strings or existing source documents from the SurrealDB database.

Where are transformation results stored?

Transformation outputs are stored as insights on the source object using the add_insight method defined in open_notebook/domain/notebook.py. This method queues a create_insight command in SurrealDB's async command queue, which later embeds the insight for semantic search while returning the result immediately to the caller.

Can I customize the system prompt for all transformations?

Yes. The DefaultPrompts model in open_notebook/domain/transformation.py provides a transformation_instructions field that prepends content to every transformation prompt. Update this singleton instance to apply global formatting requirements, such as enforcing bullet-point responses or specific tone guidelines across all transformations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}