How to Add Custom Content Transformations to the Open-Notebook Processing Pipeline

Open-Notebook enables custom content transformations through LLM-driven prompt templates that can be registered via the API and automatically executed during source ingestion without modifying core code.

Open-Notebook treats a transformation as a reusable LLM-driven text-processing step that rewrites, enriches, or filters raw content before storage. When a source such as a PDF page or web excerpt is ingested, the pipeline invokes registered transformations based on the apply_default flag stored in SurrealDB. This architecture allows you to extend the processing logic using only data-driven prompt definitions, requiring no code changes unless you need a custom UI.

What Are Content Transformations in Open-Notebook?

A transformation is a record containing a name, title, description, LLM prompt template, and an apply_default boolean flag. The flag tells the ingestion workflow whether the transformation should run automatically for every new source. The actual execution happens in open_notebook/graphs/transformation.py, which builds a LangChain chain from the stored prompt, injects the source text, runs the LLM via provision_langchain_model, and stores the output as an insight on the source record.

How to Create a Custom Transformation

Defining the Transformation Record

Create a Transformation object by specifying the prompt template and metadata. The prompt acts as the instruction set for the LLM, while apply_default controls automatic execution during ingestion.

Persisting via the API

Use the /transformations endpoints to persist the record to SurrealDB. The api/client.py wrapper provides a convenient interface:

from api.client import api_client

new_transform = api_client.create_transformation(
    name="summarize",
    title="Summarize Content",
    description="Generate a concise summary of the input text.",
    prompt="""
You are a summarizer. Return a 2‑sentence summary that preserves the main point.
""",
    apply_default=False,   # set True to run automatically on every ingest

)

print(new_transform["id"])

The api/routers/transformations.py FastAPI routes handle the underlying POST request, validating the payload against Pydantic schemas defined in api/models.py.

Executing Transformations

Running on Arbitrary Text

Execute a transformation on explicit input text using the execution endpoint, which routes through api/transformations_service.py:

result = api_client.execute_transformation(
    transformation_id="transformation:12345",
    input_text="Open‑Notebook is an open‑source research assistant …",
    model_id="openai:gpt-4o-mini",
)

print(result["output"])

# → "Open‑Notebook is…"

Integrating with the Ingestion Pipeline

To hook a transformation into the ingestion workflow, invoke the transformation graph directly from your ingestion handler:

from open_notebook.graphs.transformation import graph as transformation_graph
from open_notebook.domain.notebook import Source
from open_notebook.domain.transformation import Transformation

async def ingest_and_transform(source: Source, transformation: Transformation):
    # Run the transformation graph

    result = await transformation_graph.ainvoke(
        {
            "input_text": source.full_text,
            "source": source,
            "transformation": transformation,
        },
        config={"configurable": {"model_id": "openai:gpt-4o"}},
    )
    # result["output"] is already stored as an insight on the source

    return result["output"]

The transformation_graph in open_notebook/graphs/transformation.py automatically calls source.add_insight to persist the result when running in the context of a source.

Updating System-Wide Defaults

Edit the system-wide default instructions that govern transformation behavior via the dedicated endpoint:

api_client.update_default_prompt(
    transformation_instructions="""
You are a professional editor. Follow the style guide strictly.
"""
)

This updates the DefaultPrompts singleton defined in open_notebook/domain/transformation.py.

Key Implementation Files

• open_notebook/domain/transformation.py – Defines the Transformation record and DefaultPrompts singleton.
• open_notebook/graphs/transformation.py – Executes the LangChain chain, cleans the LLM response, and manages insight storage.
• api/transformations_service.py – Service layer wrapping HTTP calls for CRUD and execution operations.
• api/routers/transformations.py – FastAPI routes providing GET/POST/PUT/DELETE and /transformations/execute endpoints.
• api/models.py – Pydantic schemas for transformation request/response validation.
• api/client.py – Thin client wrapper used by the front-end and automation scripts.

Summary

• Data-driven architecture: Transformations are stored as records in SurrealDB, requiring no code changes to add new processing logic.
• Automatic execution: Set apply_default=True to run transformations automatically during source ingestion.
• Flexible execution: Use the API client for ad-hoc processing or invoke transformation_graph.ainvoke() directly in custom ingestion workflows.
• Insight storage: Transformation outputs are automatically saved as named insights on source records via source.add_insight.
• System defaults: Global transformation instructions can be updated via update_default_prompt without redeploying the application.

Frequently Asked Questions

What is the difference between a transformation and a notebook insight?

A transformation is a reusable LLM prompt template that processes text, while an insight is the concrete output stored on a source record. When a transformation runs, it generates an insight via source.add_insight.

Can I run multiple transformations on the same source automatically?

Yes. Enable apply_default=True for all desired transformations. The ingestion pipeline will invoke each registered transformation independently, creating multiple insights on the same source record.

Do I need to modify the source code to add a custom transformation?

No. You only need to create a Transformation record via the API or admin UI. The open_notebook/graphs/transformation.py graph handles execution dynamically based on the stored prompt template.

How does the system handle LLM model selection for transformations?

The model is specified at execution time via the configurable parameter in ainvoke() or through the model_id parameter in API calls. The provision_langchain_model function initializes the specified model provider dynamically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}