Saving and Loading CGraph Pipeline Configurations: A Complete Guide

CGraph provides built-in binary serialization via GPipeline::save() and GPipeline::load(), enabling you to persist complete pipeline configurations—including elements, events, parameters, and thread-pool settings—to disk and restore them later.

Saving and loading CGraph pipeline configurations allows you to checkpoint complex computational graphs, share pre-configured workflows, or resume long-running processes without rebuilding the pipeline from scratch. This persistence mechanism is implemented in the chunelfeng/cgraph repository using C++17 reflection capabilities and a dedicated storage subsystem.

How CGraph Pipeline Persistence Works

The persistence layer centers on two public APIs that delegate to an internal storage engine responsible for binary serialization and deserialization.

Entry Points: GPipeline::save and GPipeline::load

In src/GraphCtrl/GraphPipeline/GPipeline.cpp, the GPipeline class exposes the primary interface for saving and loading pipeline configurations.

The save method validates that no adaptor elements are present in the pipeline, then delegates to GStorage::save:

// src/GraphCtrl/GraphPipeline/GPipeline.cpp#L25-L41
CStatus GPipeline::save(const std::string& path) {
    CGRAPH_FUNCTION_BEGIN
    CGRAPH_ASSERT_INIT(false)
    for (const auto* element : repository_.elements_) {
        CGRAPH_RETURN_ERROR_STATUS_BY_CONDITION(element->isGAdaptor(),
            element->name_ + " is GAdaptor, cannot be saved.")
    }
#if __cplusplus >= 201703L
    status = GStorage::save(this, path);
#else
    status = CStatus("save function support cpp17+ only");
#endif
    CGRAPH_FUNCTION_END
}

The load method directly invokes GStorage::load to restore the pipeline state from a binary file.

The Storage Engine: GStorage Implementation

Located in src/GraphCtrl/GraphPipeline/_GStroage/GStorage.cpp, the GStorage class handles the actual serialization logic using CGraph's reflection utilities.

The save workflow follows three steps:

1. Build storage snapshot: Collects all pipeline components into a _GPipelineStorage struct
2. Serialize to buffer: Uses UReflection to convert the struct to binary
3. Write to file: Dumps the buffer to the specified path

// src/GraphCtrl/GraphPipeline/_GStroage/GStorage.cpp#L9-L18
CStatus GStorage::save(GPipelinePtr pipeline, const std::string& path) {
    CGRAPH_FUNCTION_BEGIN
    CGRAPH_ASSERT_NOT_NULL(pipeline)

    _GPipelineStorage storage {};
    status += buildPipelineStorage(pipeline, storage);
    CGRAPH_FUNCTION_CHECK_STATUS

    status += saveBuffer(storage, path);
    CGRAPH_FUNCTION_END
}

The buildPipelineStorage method iterates through the pipeline's repository and managers to populate the storage struct:

// src/GraphCtrl/GraphPipeline/_GStroage/GStorage.cpp#L54-L88
for (const auto* cur : pipeline->repository_.elements_) {
    storage.element_storages_.emplace_back(cur);
}
for (const auto& event : pipeline->event_manager_->events_map_) {
    storage.event_storages_.emplace_back(event.first, typeid(*event.second).name());
}
// ... similar loops for params, daemons, stages ...
storage.thread_pool_config_ = pipeline->schedule_.config_;

Requirements and Constraints for Saving CGraph Pipelines

Before implementing pipeline persistence, ensure your environment meets these technical requirements and constraints defined in the source code.

C++17 or Newer Required

The storage functionality relies on compile-time reflection features available only in C++17 and later. The code explicitly checks __cplusplus >= 201703L and returns an error status if compiled with an older standard.

Adaptor Elements Cannot Be Saved

The save method validates that no GAdaptor elements exist in the pipeline. Adaptors are runtime wrappers that lack complete meta-information required for serialization. If any adaptor is detected, the save operation fails with the error message "[element_name] is GAdaptor, cannot be saved."

Meta-Type Registration Required for Loading

Before calling load, you must register all custom element types that appear in the saved file using the CGRAPH_REGISTER_META_TYPE macro. This registration enables the factory to instantiate the correct concrete types during deserialization.

// tutorial/T29-Storage.cpp#L36-L40
CGRAPH_REGISTER_META_TYPE(MyNode1);
CGRAPH_REGISTER_META_TYPE(MyNode2);

Step-by-Step: Saving a CGraph Pipeline Configuration

To persist a pipeline, initialize it normally, register your elements, then call the save method with a file path.

The following example demonstrates creating a simple DAG with four nodes and saving it to "my_pipeline.cgraph":

#include <iostream>
#include "MyGNode/MyNode1.h"
#include "MyGNode/MyNode2.h"

using namespace CGraph;

void savePipeline(const std::string& path) {
    auto pipeline = GPipelineFactory::create();
    GElementPtr a, b, c, d;

    // Register elements with dependencies
    pipeline->registerGElement<MyNode1>(&a, {}, "nodeA");
    pipeline->registerGElement<MyNode2>(&b, {a}, "nodeB");
    pipeline->registerGElement<MyNode1>(&c, {a}, "nodeC");
    pipeline->registerGElement<MyNode2>(&d, {b, c}, "nodeD");

    // Save the complete configuration
    CStatus st = pipeline->save(path);
    std::cout << (st.isOK() ? "saved" : "save failed") 
              << " -> " << path << '\n';
    
    GPipelineFactory::remove(pipeline);
}

The saved binary file contains the complete pipeline state, including element types, names, dependencies, event bindings, parameters, daemon configurations, and thread-pool settings.

Restoring a CGraph Pipeline: Loading Saved Configurations

Loading reconstructs the entire pipeline from a binary file created by save. You must register all custom element types before calling load to enable the factory to instantiate the correct classes.

Registering Meta-Types

Use the CGRAPH_REGISTER_META_TYPE macro for every custom node class that appears in the saved file:

CGRAPH_REGISTER_META_TYPE(MyNode1);
CGRAPH_REGISTER_META_TYPE(MyNode2);

This registration populates the internal factory map that GStorageFactory uses during deserialization.

Loading and Executing

After registration, create a pipeline instance and call load with the file path. Once loaded, the pipeline is ready for execution via process():

void loadPipeline(const std::string& path) {
    // Register types before loading
    CGRAPH_REGISTER_META_TYPE(MyNode1);
    CGRAPH_REGISTER_META_TYPE(MyNode2);

    auto pipeline = GPipelineFactory::create();
    CStatus st = pipeline->load(path);
    
    if (st.isErr()) {
        std::cout << "load error: " << st.getInfo() << '\n';
        return;
    }
    
    // Execute the restored pipeline
    pipeline->process();
    GPipelineFactory::remove(pipeline);
    
    // Optional cleanup
    std::remove(path.c_str());
}

The load operation reconstructs the element hierarchy, restores dependencies, reapplies event bindings, and configures the thread pool according to the saved state.

Key Implementation Files in CGraph

The persistence system spans several files in the chunelfeng/cgraph repository:

File	Role
src/GraphCtrl/GraphPipeline/GPipeline.cpp	Public save and load entry points; validates adaptor restrictions
src/GraphCtrl/GraphPipeline/_GStroage/GStorage.cpp	Core serialization logic, buffer management, and file I/O
src/GraphCtrl/GraphPipeline/_GStroage/GStorageDefine.h	Struct definitions for _GPipelineStorage and component storage objects
src/GraphCtrl/GraphPipeline/_GStroage/GStorageFactory.cpp	Factory implementation for instantiating elements by type name during load
tutorial/T29-Storage.cpp	Complete working example demonstrating save/load cycles

These files implement the complete binary serialization workflow using C++17 reflection capabilities.

Summary

• CGraph pipeline configurations can be persisted to binary files using GPipeline::save() and restored with GPipeline::load().
• The feature requires C++17 or newer and is implemented in src/GraphCtrl/GraphPipeline/_GStroage/GStorage.cpp.
• Adaptor elements cannot be saved; the save method validates this via isGAdaptor() checks.
• Before loading, you must register all custom element types using CGRAPH_REGISTER_META_TYPE to enable factory instantiation.
• The saved binary includes complete pipeline state: elements, dependencies, events, parameters, daemons, stages, and thread-pool configuration.

Frequently Asked Questions

What C++ standard is required to use CGraph's save and load functionality?

C++17 or newer is mandatory. The storage system relies on compile-time reflection features that are only available when __cplusplus >= 201703L. If you compile with an older standard, both GPipeline::save and GPipeline::load will return an error status indicating that the functionality requires C++17.

Why can't I save pipelines that contain adaptor elements?

Adaptor elements are runtime wrappers that lack complete meta-information required for serialization. The GPipeline::save method explicitly checks for adaptors using element->isGAdaptor() and aborts with an error if any are found. To save a pipeline, you must use concrete element types (nodes) rather than adaptors.

How do I register custom element types before loading a saved pipeline?

Use the CGRAPH_REGISTER_META_TYPE macro for each custom class that appears in the saved file. This registration must occur before calling GPipeline::load(). For example:

CGRAPH_REGISTER_META_TYPE(MyNode1);
CGRAPH_REGISTER_META_TYPE(MyNode2);

This populates the internal factory map that GStorageFactory uses to instantiate the correct concrete types during deserialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}