Create Custom Content Transformation Processing Sources with AI in Open Notebook

Open Notebook enables users to create reusable AI-driven transformations that process any source content through a declarative pipeline combining SurrealDB storage, LangGraph execution, and multi-provider LLM support.

You can create custom content transformation processing sources with AI by defining reusable prompt templates that automatically process PDFs, webpages, and audio files through a structured LangGraph pipeline. The transformation system in Open Notebook treats each custom prompt as a first-class entity stored in SurrealDB, allowing you to apply consistent AI processing across diverse content types. This architecture separates the transformation definition from execution, enabling non-technical users to craft prompts while the backend handles complex model provisioning and graph orchestration.

Understanding the Transformation Architecture

A transformation in Open Notebook is fundamentally a stored AI prompt combined with execution logic. The system persists transformation definitions as records containing a name, title, description, and the LLM prompt template. When executed, the transformation processes source content through a dedicated LangGraph that handles model selection, prompt composition, and result storage.

The architecture relies on four core components working in sequence:

• Domain Model (open_notebook/domain/transformation.py) - Defines the Transformation class and DefaultPrompts singleton for global instruction management
• API Router (api/routers/transformations.py) - Exposes HTTP endpoints for CRUD operations and execution via FastAPI
• Graph Execution (open_notebook/graphs/transformation.py) - Implements the LangGraph that orchestrates the LLM interaction
• Service Layer (api/transformations_service.py) - Provides a Pythonic wrapper around the HTTP client for programmatic access

Define a Custom Transformation

Transformations are stored as database records in SurrealDB through the Transformation domain model. Each record encapsulates the prompt template and metadata required for reusable AI processing.

The Transformation class in open_notebook/domain/transformation.py provides the persistence layer, while the API router in api/routers/transformations.py handles HTTP requests to create these records. When you create a transformation, you specify the prompt that will be sent to the LLM along with optional model overrides.

Persisting Transformation Definitions

The backend exposes REST endpoints for transformation management. The Transformation.save() method persists the record to SurrealDB, making the transformation available for execution across the application. The service layer wraps these operations for convenient Python usage.

Execute AI Transformations with LangGraph

Execution occurs through a dedicated LangGraph defined in open_notebook/graphs/transformation.py. This graph contains a single node run_transformation that orchestrates the entire AI processing pipeline.

The Transformation Graph Flow

The execution flow follows a precise sequence:

1. Prompt Composition - The system merges the stored transformation prompt with optional default instructions from the DefaultPrompts singleton
2. Model Provisioning - provision_langchain_model (from open_notebook/ai/models.py) instantiates the correct LLM using the Esperanto multi-provider manager
3. Content Processing - The graph sends the composed prompt plus source content to the provisioned model
4. Output Cleaning - clean_thinking_content removes any "thinking" tokens or extraneous formatting from the LLM response
5. Result Storage - The cleaned output is stored as an insight on the original source, making it searchable alongside the raw content

The graph is invoked asynchronously using transformation_graph.ainvoke, ensuring non-blocking execution for long-running AI operations.

Programmatic Usage Examples

You can interact with the transformation system through multiple interfaces: the Python service layer, direct API calls, or the TypeScript frontend client.

Creating Transformations via the Service Layer

Use TransformationsService to create reusable transformation templates programmatically:

from api.transformations_service import transformations_service

new_trans = transformations_service.create_transformation(
    name="summarize",
    title="Summarize Content",
    description="Generates a concise summary of the source text.",
    prompt="You are a helpful assistant. Summarize the following content in 3 sentences:",
    apply_default=True,
)
print(new_trans.id, new_trans.title)

This service layer method corresponds to the POST /transformations endpoint in api/routers/transformations.py.

Running Transformations on Source Content

Execute a stored transformation against any text content using the service layer:

result = transformations_service.execute_transformation(
    transformation_id="transform-123",
    input_text="Full text extracted from a PDF …",
    model_id=None,  # uses the default model for the transformation

)
print(result["output"])

Under the hood, this triggers transformation_graph.ainvoke with the transformation ID and input text, routing through the LangGraph defined in open_notebook/graphs/transformation.py.

Managing Global Default Prompts

The DefaultPrompts singleton stores global instructions that automatically prefix every transformation unless explicitly disabled. Update these defaults through the API client:

from api.client import api_client

api_client.put_default_prompt({
    "transformation_instructions": "Always keep the tone formal and avoid bullet points."
})

This corresponds to the PUT /transformations/default-prompt endpoint, which modifies the base instructions applied to all transformations.

Frontend Integration

The TypeScript API wrapper in frontend/src/lib/api/transformations.ts mirrors the backend routes, enabling React components to manage transformations directly:

import { transformationsApi } from '@/lib/api/transformations';

// List all transformations
const list = await transformationsApi.list();

// Execute one
const execRes = await transformationsApi.execute({
  transformation_id: list[0].id,
  input_text: source.fullText,
});
console.log(execRes.output);

This frontend integration maps directly to the FastAPI routes, ensuring type safety and consistent behavior across the stack.

Summary

• Transformations are declarative AI prompts stored as SurrealDB records via the Transformation class in open_notebook/domain/transformation.py
• Execution flows through LangGraph in open_notebook/graphs/transformation.py, specifically the run_transformation node that handles prompt composition and model provisioning
• Multi-provider support comes from the Esperanto model manager in open_notebook/ai/models.py, enabling flexible LLM selection per transformation
• Global defaults can be configured via the DefaultPrompts singleton to ensure consistent AI behavior across all transformations
• Full-stack accessibility is provided through the Python service layer (api/transformations_service.py) and TypeScript frontend client (frontend/src/lib/api/transformations.ts)

Frequently Asked Questions

What is a transformation in Open Notebook?

A transformation is a reusable AI-driven prompt template that processes source content through a structured pipeline. According to the Open Notebook source code, it is defined as a Transformation record containing a name, title, description, and the LLM prompt. These records are stored in SurrealDB and can be applied to any source type including PDFs, webpages, and audio files.

How does the default prompt system work?

The system uses a singleton DefaultPrompts record stored in open_notebook/domain/transformation.py to maintain global instructions. When executing a transformation with apply_default=True, the graph automatically prefixes these global instructions to the specific transformation prompt. You can modify the default prompt via the PUT /transformations/default-prompt endpoint, affecting all subsequent transformations unless they explicitly override the behavior.

Can I use different AI models for different transformations?

Yes, each transformation record supports an optional model override. When calling execute_transformation, you can specify a model_id parameter to use a specific LLM, or pass None to use the system default. The provision_langchain_model function in open_notebook/ai/models.py handles the actual model instantiation through the Esperanto multi-provider manager, supporting various LLM backends.

Where are transformation results stored?

After the LangGraph processes the content through run_transformation and cleans the output using clean_thinking_content, the results are stored as insights on the original source record in SurrealDB. This makes transformed content searchable alongside the original source material, maintaining the relationship between raw input and AI-processed output within the Open Notebook data model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}