# How Open Notebook Applies Transformations to Sources Using the transformation.py Graph

> Learn how Open Notebook applies LLM transformations to sources using the transformation.py graph. Discover how it resolves inputs, builds prompts, calls the LLM, and saves results.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-27

---

**Open Notebook applies user-defined LLM transformations to sources by executing a compiled LangGraph state machine in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) that resolves inputs, builds prompts, calls the LLM, and persists results as insights.**

Open Notebook enables intelligent document processing through a dedicated transformation system that executes user-defined prompts against source content. The core orchestration happens in the **[`transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/transformation.py) graph**, a minimal LangGraph implementation that handles the entire lifecycle from input validation to insight persistence. This architecture leverages FastAPI's async capabilities and SurrealDB's command queue to provide a responsive, fire-and-forget transformation pipeline.

## The Transformation Execution Architecture

The transformation process follows a linear pipeline that begins with an HTTP request and terminates with persisted insights. Understanding this flow is essential for debugging custom transformations or extending the system.

### API Entry Point and Graph Invocation

Transformations are triggered through the REST endpoint **`POST /transformations/execute`** defined in [`api/routers/transformations.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/transformations.py). This endpoint validates the requested `Transformation` object and target model, then invokes the compiled graph asynchronously:

```python
result = await transformation_graph.ainvoke(
    {"input_text": request.input_text,
     "transformation": transformation},
    config={"configurable": {"model_id": request.model_id}},
)

```

The `transformation_graph` object is the compiled LangGraph instance exported from [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py).

### Graph Structure and Compilation

The transformation graph is a minimal **StateGraph** compiled at module import time via `graph = agent_state.compile()`. The topology consists of a single execution node:

```text
START → agent → END

```

The `agent` node executes the `run_transformation` function, which contains the core business logic. This design ensures predictable, linear execution without conditional branching or cycles, making the transformation process straightforward to trace and debug.

## The run_transformation Node Implementation

The `run_transformation` function in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) performs six distinct operations to process source content:

### 1. Input Resolution

The node first retrieves the `source` object and `content` from the graph state. If `input_text` is not provided, it automatically falls back to `source.full_text`:

```python
source_obj = state.get("source")
content = state.get("input_text") or source_obj.full_text

```

This ensures transformations can operate on either explicit text inputs or existing source documents.

### 2. System Prompt Construction

The node builds the final prompt by combining the transformation's template with global instructions. If `DefaultPrompts.transformation_instructions` exists, it prepends these instructions to the user-defined prompt, followed by the delimiter `# INPUT`:

```python
system_prompt = Prompter(template_text=transformation_template_text).render(data=state)

```

### 3. LLM Invocation via Esperanto

The graph utilizes the **`provision_langchain_model`** helper to instantiate a LangChain chain configured with the `model_id` supplied via `configurable.model_id`. The payload consists of a `SystemMessage` containing the constructed prompt and a `HumanMessage` containing the source text:

```python
chain = await provision_langchain_model(...)
response = await chain.ainvoke(payload)

```

### 4. Output Cleaning

Raw LLM responses are processed through `extract_text_content` and `clean_thinking_content` to remove artifacts, thinking tokens, and formatting noise, yielding plain text suitable for storage.

### 5. Insight Persistence

If a `Source` object was provided in the state, the cleaned output is persisted as an insight using `source.add_insight(transformation.title, cleaned_content)`. This method queues a `create_insight` command in SurrealDB's async command queue, enabling fire-and-forget persistence without blocking the response:

```python
await source.add_insight(transformation.title, cleaned_content)

```

### 6. Result Propagation

Finally, the node returns a dictionary containing the cleaned output, which propagates back to the API caller:

```python
return {"output": cleaned_content}

```

## Practical Implementation Examples

### Calling the Transformation Endpoint

Execute a transformation via the REST API by posting to the transformations router:

```python
import httpx

payload = {
    "input_text": "Summarize the key findings of this PDF.",
    "transformation_id": "transformation:abc123",
    "model_id": "model:openai-gpt-4o"
}
resp = httpx.post("http://localhost:5055/transformations/execute", json=payload)
print(resp.json()["output"])

```

This returns the LLM-generated content and, if a source was specified, stores the result as an insight.

### Direct Graph Invocation

For programmatic access within the Open Notebook codebase, import and invoke the graph directly:

```python
from open_notebook.graphs.transformation import graph as transformation_graph
from open_notebook.domain.transformation import Transformation
from open_notebook.domain.notebook import Source

state = {
    "input_text": None,  # Uses source.full_text automatically

    "source": source,
    "transformation": transformation,
}
config = {"configurable": {"model_id": "model:anthropic-claude-3"}}

result = await transformation_graph.ainvoke(state, config=config)
print("Insight generated:", result["output"])

```

### Customizing Default Instructions

Modify global transformation behavior by updating the `DefaultPrompts` singleton:

```python
from open_notebook.domain.transformation import DefaultPrompts

default = await DefaultPrompts.get_instance()
default.transformation_instructions = "Always answer in bullet points."
await default.update()

```

This prepends the instruction to all subsequent transformation prompts.

## Summary

- **Open Notebook** applies transformations through a compiled LangGraph in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) with a linear START → agent → END topology.
- The **`run_transformation`** node resolves inputs, constructs system prompts using `Prompter`, and invokes LLMs via `provision_langchain_model`.
- Results are cleaned using `clean_thinking_content` and persisted via `source.add_insight`, which queues async SurrealDB commands.
- The **REST endpoint** `POST /transformations/execute` in [`api/routers/transformations.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/transformations.py) provides the primary entry point for external clients.
- **Global instructions** can be configured through the `DefaultPrompts` model to prepend standard formatting to all transformations.

## Frequently Asked Questions

### What is the transformation.py graph in Open Notebook?

The [`transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/transformation.py) graph is a minimal LangGraph StateGraph defined in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) that orchestrates LLM-based text transformations. It consists of a single `agent` node executing the `run_transformation` function, compiled at import time to handle the complete workflow from prompt construction to insight persistence.

### How does the transformation graph handle source content?

The graph accepts source content through the `input_text` state field or automatically resolves it from `source.full_text` if the field is empty. This dual-input design allows the transformation system to process either ad-hoc text strings or existing source documents from the SurrealDB database.

### Where are transformation results stored?

Transformation outputs are stored as **insights** on the source object using the `add_insight` method defined in [`open_notebook/domain/notebook.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/notebook.py). This method queues a `create_insight` command in SurrealDB's async command queue, which later embeds the insight for semantic search while returning the result immediately to the caller.

### Can I customize the system prompt for all transformations?

Yes. The `DefaultPrompts` model in [`open_notebook/domain/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/transformation.py) provides a `transformation_instructions` field that prepends content to every transformation prompt. Update this singleton instance to apply global formatting requirements, such as enforcing bullet-point responses or specific tone guidelines across all transformations.