How Content Transformations Process Sources Using Custom Prompts in Open Notebook

Open Notebook processes sources using custom prompts through a state-driven LangGraph pipeline that assembles user-defined instructions with source content, executes them via an LLM, and persists the results as structured insights.

The lfnovo/open-notebook repository implements a reusable transformation system that converts raw source material into structured notes using configurable LLM prompts. This architecture allows users to define custom analysis patterns once and apply them automatically to any number of sources. Understanding how content transformations process sources using custom prompts reveals a sophisticated orchestration between graph-based state management, prompt assembly, and response cleaning.

Understanding the Transformation Pipeline

The core processing logic resides in open_notebook/graphs/transformation.py, which implements a Transformation graph using LangGraph. This graph coordinates the entire lifecycle from prompt preparation to insight persistence.

State Preparation with TransformationState

When a transformation is invoked, the system creates a TransformationState typed dictionary containing three critical elements: the source object, the raw input text (if provided), and the Transformation record that stores the custom prompt. The state definition in open_notebook/graphs/transformation.py establishes the contract for all subsequent operations in the pipeline.

Prompt Assembly and Default Instructions

The prompt construction process follows a specific hierarchy. First, the system retrieves the stored transformation.prompt from the Transformation model defined in open_notebook/domain/transformation.py. If a global default instruction exists (DefaultPrompts.transformation_instructions), the system prefixes it to the custom prompt, allowing administrators to enforce common formatting standards across all transformations. The final assembled prompt appends an # INPUT marker followed by the source content or supplied input_text as a human message, creating a complete instruction set for the LLM.

Model Execution and Response Processing

Once the prompt is assembled, the graph transitions to model provisioning and execution phases.

LLM Provisioning via Esperanto

The assembled prompt is passed to provision_langchain_model, which selects an appropriate LLM provider through the Esperanto library and returns a LangChain chain configured for async invocation. This abstraction layer allows the system to support multiple model providers without modifying the core transformation logic.

Response Cleaning and Insight Creation

After the LLM returns a response via await chain.ainvoke(payload), the raw output undergoes processing through two specialized functions. The extract_text_content function strips non-text elements and artifacts, while clean_thinking_content removes internal reasoning markers (such as citations or code blocks), leaving only the final user-visible output. If a source object is present in the state, the cleaned content is persisted as an insight attached to that source through await source.add_insight(transformation.title, cleaned_content).

Service Layer and API Integration

The service layer in api/transformations_service.py exposes CRUD endpoints for transformation management and provides the execute_transformation method. This method forwards requests to the API client, which ultimately triggers the LangGraph pipeline described above. All errors are normalized through classify_error and re-raised as OpenNotebookError to maintain API consistency across the platform.

Frontend Implementation

The React frontend interfaces with the transformation system through frontend/src/lib/api/transformations.ts and frontend/src/lib/types/transformations.ts. These TypeScript modules provide type-safe wrappers that allow users to create custom transformations, select sources, and launch processing jobs without writing backend code. The frontend handles the complete user flow from prompt creation to result display.

Code Examples

Creating a Custom Transformation

The following Python code demonstrates creating a reusable transformation with a structured custom prompt:

from api.transformations_service import transformations_service

trans = transformations_service.create_transformation(
    name="paper_analysis",
    title="Paper Analysis",
    description="Extract key sections from academic papers",
    prompt="""
        Analyze the given paper and extract:
        1. Research Question
        2. Hypothesis
        3. Methodology
        4. Key Findings (as a numbered list)
        5. Limitations
        6. Future Work
        Provide each item under a clear heading.
    """,
    apply_default=False,
)
print(f"Created transformation {trans.id}")

Executing Transformations Programmatically

Execute the transformation against a source using the service layer:

result = transformations_service.execute_transformation(
    transformation_id=trans.id,
    input_text=None,           # Use the source's full_text automatically

    model_id="gpt-4o-mini",   # Choose any registered model

)

# `result` contains the cleaned note content returned from the graph

print(result["output"])

Frontend Integration with React

Implement transformations in the frontend using the provided React hooks:

import { useTransformations } from '@/lib/api/transformations';

const { createTransformation, executeTransformation } = useTransformations();

async function run() {
  const newTrans = await createTransformation({
    name: 'summary',
    title: 'Summary',
    description: 'Brief 200‑300 word overview',
    prompt: 'Summarize the source in 200‑300 words.',
    apply_default: true,
  });

  const output = await executeTransformation({
    transformationId: newTrans.id,
    inputText: '',          // empty – graph falls back to source.full_text
    modelId: 'openai:gpt-4o',
  });

  console.log(output);
}

Summary

• State-driven architecture: The TransformationState in open_notebook/graphs/transformation.py coordinates all data flow between prompt preparation and insight storage.

• Hierarchical prompt assembly: Custom prompts combine with optional default instructions, then append source content marked with # INPUT for clear LLM instruction boundaries.

• Multi-stage processing: The pipeline provisions models via Esperanto, executes async LLM calls, and cleans responses using extract_text_content and clean_thinking_content.

• Persistent insights: Results are automatically attached to source objects as named insights, creating a searchable knowledge base.

• Full-stack integration: Service layers and frontend TypeScript wrappers provide complete API access without requiring direct graph manipulation.

Frequently Asked Questions

What is the role of the TransformationState in Open Notebook?

The TransformationState is a typed dictionary defined in open_notebook/graphs/transformation.py that serves as the single source of truth for the transformation pipeline. It contains the source object, raw input text, and the Transformation record holding the custom prompt. This state object is passed between each node in the LangGraph, ensuring that all processing steps have access to the necessary context without external dependencies.

How does Open Notebook handle custom prompts versus default instructions?

The system checks for the existence of DefaultPrompts.transformation_instructions before executing a transformation. If present, these default instructions are prefixed to the user's custom prompt stored in transformation.prompt. This hierarchical approach allows administrators to enforce universal formatting rules (such as citation styles or output structures) while preserving user-specific analysis instructions. The combined prompt is then appended with the # INPUT marker and source content.

Where are transformation results stored after processing?

Cleaned transformation outputs are stored as insights attached directly to the source object. Specifically, the graph calls await source.add_insight(transformation.title, cleaned_content) in open_notebook/graphs/transformation.py, which persists the result with the transformation title as the insight name. This creates a permanent relationship between the analysis result and the original source material, accessible through the source's insight collection.

What error handling mechanisms exist in the transformation pipeline?

All errors encountered during graph execution are captured and normalized through the classify_error function, then re-raised as OpenNotebookError instances. This standardization occurs in open_notebook/graphs/transformation.py and ensures that API consumers receive consistent error formats regardless of whether the failure originated from LLM providers, model provisioning, or response cleaning operations. The OpenNotebookError type provides structured error information suitable for programmatic handling by frontend clients.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}