# How to Add Custom Content Transformations to the Open-Notebook Processing Pipeline

> Easily add custom content transformations to the Open Notebook processing pipeline using LLM-driven prompt templates registered via API. Enhance ingestion without core code changes.

- Repository: [Luis Novo/open-notebook](https://github.com/lfnovo/open-notebook)
- Tags: how-to-guide
- Published: 2026-06-22

---

**Open-Notebook enables custom content transformations through LLM-driven prompt templates that can be registered via the API and automatically executed during source ingestion without modifying core code.**

Open-Notebook treats a *transformation* as a reusable LLM-driven text-processing step that rewrites, enriches, or filters raw content before storage. When a source such as a PDF page or web excerpt is ingested, the pipeline invokes registered transformations based on the `apply_default` flag stored in SurrealDB. This architecture allows you to extend the processing logic using only data-driven prompt definitions, requiring no code changes unless you need a custom UI.

## What Are Content Transformations in Open-Notebook?

A **transformation** is a record containing a name, title, description, LLM prompt template, and an `apply_default` boolean flag. The flag tells the ingestion workflow whether the transformation should run automatically for every new source. The actual execution happens in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py), which builds a LangChain chain from the stored prompt, injects the source text, runs the LLM via `provision_langchain_model`, and stores the output as an insight on the source record.

## How to Create a Custom Transformation

### Defining the Transformation Record

Create a `Transformation` object by specifying the prompt template and metadata. The prompt acts as the instruction set for the LLM, while `apply_default` controls automatic execution during ingestion.

### Persisting via the API

Use the `/transformations` endpoints to persist the record to SurrealDB. The [`api/client.py`](https://github.com/lfnovo/open-notebook/blob/main/api/client.py) wrapper provides a convenient interface:

```python
from api.client import api_client

new_transform = api_client.create_transformation(
    name="summarize",
    title="Summarize Content",
    description="Generate a concise summary of the input text.",
    prompt="""
You are a summarizer. Return a 2‑sentence summary that preserves the main point.
""",
    apply_default=False,   # set True to run automatically on every ingest

)

print(new_transform["id"])

```

The [`api/routers/transformations.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/transformations.py) FastAPI routes handle the underlying `POST` request, validating the payload against Pydantic schemas defined in [`api/models.py`](https://github.com/lfnovo/open-notebook/blob/main/api/models.py).

## Executing Transformations

### Running on Arbitrary Text

Execute a transformation on explicit input text using the execution endpoint, which routes through [`api/transformations_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/transformations_service.py):

```python
result = api_client.execute_transformation(
    transformation_id="transformation:12345",
    input_text="Open‑Notebook is an open‑source research assistant …",
    model_id="openai:gpt-4o-mini",
)

print(result["output"])

# → "Open‑Notebook is…"

```

### Integrating with the Ingestion Pipeline

To hook a transformation into the ingestion workflow, invoke the transformation graph directly from your ingestion handler:

```python
from open_notebook.graphs.transformation import graph as transformation_graph
from open_notebook.domain.notebook import Source
from open_notebook.domain.transformation import Transformation

async def ingest_and_transform(source: Source, transformation: Transformation):
    # Run the transformation graph

    result = await transformation_graph.ainvoke(
        {
            "input_text": source.full_text,
            "source": source,
            "transformation": transformation,
        },
        config={"configurable": {"model_id": "openai:gpt-4o"}},
    )
    # result["output"] is already stored as an insight on the source

    return result["output"]

```

The `transformation_graph` in [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) automatically calls `source.add_insight` to persist the result when running in the context of a source.

## Updating System-Wide Defaults

Edit the system-wide default instructions that govern transformation behavior via the dedicated endpoint:

```python
api_client.update_default_prompt(
    transformation_instructions="""
You are a professional editor. Follow the style guide strictly.
"""
)

```

This updates the `DefaultPrompts` singleton defined in [`open_notebook/domain/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/transformation.py).

## Key Implementation Files

- **[`open_notebook/domain/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/domain/transformation.py)** – Defines the `Transformation` record and `DefaultPrompts` singleton.
- **[`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py)** – Executes the LangChain chain, cleans the LLM response, and manages insight storage.
- **[`api/transformations_service.py`](https://github.com/lfnovo/open-notebook/blob/main/api/transformations_service.py)** – Service layer wrapping HTTP calls for CRUD and execution operations.
- **[`api/routers/transformations.py`](https://github.com/lfnovo/open-notebook/blob/main/api/routers/transformations.py)** – FastAPI routes providing `GET/POST/PUT/DELETE` and `/transformations/execute` endpoints.
- **[`api/models.py`](https://github.com/lfnovo/open-notebook/blob/main/api/models.py)** – Pydantic schemas for transformation request/response validation.
- **[`api/client.py`](https://github.com/lfnovo/open-notebook/blob/main/api/client.py)** – Thin client wrapper used by the front-end and automation scripts.

## Summary

- **Data-driven architecture**: Transformations are stored as records in SurrealDB, requiring no code changes to add new processing logic.
- **Automatic execution**: Set `apply_default=True` to run transformations automatically during source ingestion.
- **Flexible execution**: Use the API client for ad-hoc processing or invoke `transformation_graph.ainvoke()` directly in custom ingestion workflows.
- **Insight storage**: Transformation outputs are automatically saved as named insights on source records via `source.add_insight`.
- **System defaults**: Global transformation instructions can be updated via `update_default_prompt` without redeploying the application.

## Frequently Asked Questions

### What is the difference between a transformation and a notebook insight?

A transformation is a reusable LLM prompt template that processes text, while an insight is the concrete output stored on a source record. When a transformation runs, it generates an insight via `source.add_insight`.

### Can I run multiple transformations on the same source automatically?

Yes. Enable `apply_default=True` for all desired transformations. The ingestion pipeline will invoke each registered transformation independently, creating multiple insights on the same source record.

### Do I need to modify the source code to add a custom transformation?

No. You only need to create a `Transformation` record via the API or admin UI. The [`open_notebook/graphs/transformation.py`](https://github.com/lfnovo/open-notebook/blob/main/open_notebook/graphs/transformation.py) graph handles execution dynamically based on the stored prompt template.

### How does the system handle LLM model selection for transformations?

The model is specified at execution time via the `configurable` parameter in `ainvoke()` or through the `model_id` parameter in API calls. The `provision_langchain_model` function initializes the specified model provider dynamically.