How Open Notebook Executes Content Transformations Using LangGraph State Machines and ai_prompter Templates

Open Notebook runs content transformations as compiled LangGraph workflows that leverage ai_prompter to render Jinja templates, provision LangChain models dynamically, and return structured outputs through a single-node execution graph.

The lfnovo/open-notebook repository implements content transformations as reusable, asynchronous workflows. By combining LangGraph state machines for orchestration and ai_prompter for flexible template rendering, the system transforms raw text into structured insights through a deterministic execution model defined in open_notebook/graphs/transformation.py.

Defining the Typed State Schema

The transformation workflow begins with a strictly typed state definition. In open_notebook/graphs/transformation.py (lines 16-21), the TransformationState TypedDict encapsulates all mutable data flowing through the graph:

• input_text: The raw content to be transformed
• source: The optional source record providing context
• transformation: The transformation definition containing the prompt template and metadata
• output: The final transformed text returned by the LLM

This schema ensures type safety across the asynchronous execution boundary while maintaining flexibility for different transformation types.

The Transformation Execution Node

The core logic resides in the run_transformation node function (lines 24-60). This single node orchestrates the entire LLM interaction pipeline through five distinct phases:

1. Source Text Resolution: The node extracts the target content, preferring explicit input_text over the source record's content (lines 24-33).

2. Prompt Template Construction: It builds a Jinja-style system prompt by combining optional default instructions with the transformation's stored prompt field (lines 34-38).

3. ai_prompter Rendering: The template is rendered using ai_prompter via Prompter(template_text=...).render(data=state) (lines 40-42), injecting the current state dictionary into the Jinja template.

4. Model Provisioning: The node creates a LangChain chain dynamically via provision_langchain_model (lines 45-49), supporting model selection through the configurable model_id parameter.

5. Response Processing: After receiving the LLM response, the node extracts plain text, strips reasoning artifacts using clean_thinking_content, and stores the result as an insight on the source record (lines 52-60).

Compiling the State Graph

Once the node is defined, the system wraps it in a StateGraph with a minimal execution path. In open_notebook/graphs/transformation.py (lines 71-75), the graph is compiled with a single edge from START to agent (the transformation node) and then to END. This creates a deterministic, reproducible workflow that handles async execution and error boundaries automatically.

API Integration and Service Layer

When clients call POST /transformations/{id}/execute, the request flows through api/transformations_service.py (lines 106-114). This service layer retrieves the stored transformation definition, forwards the request to the compiled graph with the optional model_id and input_text passed through the configurable context, and returns the transformed content as a domain object.

Practical Implementation Examples

You can execute transformations through three interfaces depending on your integration needs.

Direct Graph Invocation

For maximum control within Python applications, invoke the compiled graph directly:

from open_notebook.graphs.transformation import graph
from open_notebook.domain.transformation import Transformation
from open_notebook.domain.notebook import Source

state = {
    "input_text": "The quick brown fox jumps over the lazy dog.",
    "source": Source(...),               # optional, can be None

    "transformation": Transformation(
        name="summarize",
        title="Summarize Text",
        description="Creates a short summary.",
        prompt="Summarize the following text:",
        apply_default=False,
    ),
}

# Execute the graph; `config` can carry a model id for the LLM

result = await graph.ainvoke(state, config={"configurable": {"model_id": "gpt-4o"}})
print(result["output"])

High-Level Service Interface

For standard operations, use the service layer abstraction:

from open_notebook.services import transformations_service

# Execute an existing transformation stored in the DB

output = transformations_service.execute_transformation(
    transformation_id="tr_12345",
    input_text="Explain quantum entanglement in plain language.",
    model_id="anthropic/claude-3.5-sonnet"
)
print(output)   # → transformed text returned from the LLM

REST API Endpoint

For external integrations, call the HTTP endpoint:

curl -X POST http://localhost:5055/transformations/tr_12345/execute \
     -H "Content-Type: application/json" \
     -d '{"input_text":"Explain quantum entanglement","model_id":"gpt-4o"}'

Key Implementation Files

The transformation system spans several critical modules:

• open_notebook/graphs/transformation.py: Contains the LangGraph state definition, run_transformation node, and graph compilation logic
• open_notebook/domain/transformation.py: Defines the Pydantic model storing transformation prompts and metadata
• api/transformations_service.py: Service layer bridging HTTP requests to graph execution
• open_notebook/ai/provision.py: Helper module creating LangChain chains for selected models
• prompts/: Directory storing reusable Jinja templates referenced by transformation definitions

Summary

• LangGraph orchestrates transformations as compiled state machines with a single execution node and deterministic flow from START to END.
• ai_prompter renders Jinja templates dynamically, allowing transformations to inject state variables into system prompts via Prompter.render().
• The run_transformation node in transformation.py handles the complete LLM lifecycle: text extraction, prompt building, model provisioning via provision_langchain_model, and response cleaning with clean_thinking_content.
• The API layer in transformations_service.py provides both direct graph access and high-level service abstractions for HTTP clients.
• All transformation outputs are stored as insights on source records, maintaining provenance between raw content and LLM-generated derivatives.

Frequently Asked Questions

How does ai_prompter integrate with the LangGraph state machine?

The run_transformation node calls Prompter(template_text=...).render(data=state) (lines 40-42 in transformation.py) to process Jinja templates stored in the transformation definition. This occurs after the node builds the system prompt and before provisioning the LangChain model, allowing the state dictionary to dynamically populate template variables.

What is the execution flow when calling the transformation API endpoint?

When POST /transformations/{id}/execute receives a request, transformations_service.py (lines 106-114) retrieves the transformation definition, packages the input_text and optional model_id into the graph configuration, and invokes the compiled graph. The state machine executes asynchronously, and the service returns the transformed content once the run_transformation node completes its LLM interaction and response processing.

Can I use custom model providers with the transformation graph?

Yes. The provision_langchain_model function in open_notebook/ai/provision.py creates LangChain chains dynamically based on the model_id passed through the graph's configurable state. This supports any model provider compatible with the provision module, including OpenAI GPT models and Anthropic Claude instances, specified via the model_id parameter in direct invocation or API calls.

How does the system handle LLM responses containing reasoning artifacts?

The run_transformation node applies clean_thinking_content (lines 52-60 in transformation.py) to strip reasoning artifacts from the raw LLM response before storing the result. This ensures that only the relevant transformed content is saved as an insight on the source record, removing internal "thinking" or chain-of-thought content that some models include in their outputs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}