Pipeline Configuration Serialization and Deserialization in ApraPipes

ApraPipes handles pipeline configuration serialization and deserialization through an intermediate representation (IR) called PipelineDescription, enabling round-trip conversion between JSON documents and C++ data structures via the JsonParser class and the toJson method.

The apra-labs/aprapipes repository implements a type-safe system for pipeline configuration serialization and deserialization that decouples the persistence format from the runtime engine. By representing pipelines as a hierarchy of plain C++ structs, the framework supports JSON as a transport and debugging format while maintaining strict in-memory data models for modules, connections, and global settings.

The Intermediate Representation (IR) Architecture

ApraPipes defines its pipeline IR through a set of simple structs that aggregate configuration data without any JSON-specific logic.

Core Data Structures

The PipelineDescription struct acts as the root container and is composed of three primary elements:

• PipelineSettings: Stores global options including name, version, description, queue_size, on_error, and auto_start.
• ModuleInstance: Represents a single module with instance_id, module_type, and a properties map of type std::map<std::string, PropertyValue>.
• Connection: Defines wires between pins using from_module, from_pin, to_module, to_pin, and an optional sieve flag.

The PropertyValue type is implemented as a std::variant capable of holding scalar types (int64_t, double, bool, std::string) or homogeneous arrays of those scalars. This design allows the same IR to support multiple configuration formats without modification.

Deserializing Pipeline Configurations from JSON

The JsonParser class in base/src/declarative/JsonParser.cpp implements the complete deserialization pipeline, converting JSON inputs into populated PipelineDescription objects.

Entry Points and Initial Processing

Deserialization begins through two static entry points:

• JsonParser::parseFile(const std::string& path) loads JSON from disk using the nlohmann/json library.
• JsonParser::parseString(const std::string& json, const std::string& sourceName) parses inline JSON strings.

Both methods forward the root JSON object to a shared parse helper that returns a ParseResult struct containing the populated description and an optional error string.

Section-by-Section Parsing Logic

The parser delegates processing to four specialized static functions that walk specific JSON sections:

1. parsePipelineSection: Extracts top-level metadata fields such as name, version, and description from the pipeline object.
2. parseSettingsSection: Reads global configuration under settings, supporting both camelCase (queueSize) and snake_case (queue_size) keys for parameters like error handling and auto-start behavior.
3. parseModulesSection: Iterates over the modules object, creating a ModuleInstance for each key. It extracts the mandatory "type" field and optional "props" map, converting each property value to PropertyValue via the toPropertyValue helper.
4. parseConnectionsSection: Validates the connections array, extracts "from" and "to" strings, and parses the module.pin syntax using Connection::parse. It optionally reads a sieve boolean flag.

Module Registration and Error Handling

Before parsing, the framework calls ensureBuiltinModulesRegistered() to guarantee that built-in module types are available for validation. Throughout the process, errors throw std::runtime_error with descriptive messages that are caught and aggregated into the ParseResult.error field, allowing the caller to distinguish between success and failure states without crashing.

Serializing Pipeline Configurations to JSON

The PipelineDescription::toJson method in base/src/declarative/PipelineDescription.cpp handles serialization, manually constructing a JSON string that mirrors the format accepted by the parser.

Struct-to-JSON Conversion

The serialization process walks the IR and emits formatted output:

• Settings Block: Writes the PipelineSettings struct using escapeJson for string safety.
• Source Block: Emits optional source_format and source_path fields.
• Modules Array: For each ModuleInstance, serializes instance_id, module_type, and the properties map using propertyValueToString. This helper formats scalars, quoted strings, and array literals ([ ... ]) appropriately.
• Connections Array: Serializes each Connection by concatenating module.pin strings, omitting the pin suffix if empty.

The resulting string is a valid JSON document suitable for debugging, network transport, or persistence to disk.

Practical Workflow Examples

Loading and Modifying a Pipeline

The following example demonstrates loading a configuration, mutating a property, and re-serializing:

#include "declarative/JsonParser.h"
#include "declarative/PipelineDescription.h"

int main() {
    // Deserialize from file
    auto result = apra::JsonParser::parseFile("examples/basic/split_pipeline.json");
    if (!result.success) {
        std::cerr << "Parse error: " << result.error << '\n';
        return 1;
    }

    // Modify the in-memory IR
    auto* decoder = result.description.findModule("my_decoder");
    if (decoder) {
        decoder->properties["bitrate"] = int64_t(4000000);
    }

    // Serialize back to JSON
    std::string json = result.description.toJson();
    std::cout << json << std::endl;
}

Parsing Inline Configuration

For embedded configurations, use parseString:

std::string json = R"({
  "pipeline": { "name": "demo", "version": "1.0" },
  "settings": { "queueSize": 5, "autoStart": true },
  "modules": {
    "source": { "type": "FileSource", "props": { "path": "video.mp4" } }
  },
  "connections": [{ "from": "source", "to": "decoder" }]
})";

auto result = apra::JsonParser::parseString(json, "<inline>");
if (result.success) {
    std::cout << "Loaded: " << result.description.settings.name << '\n';
}

Programmatic Construction and Export

You can also build pipelines in code and export them:

apra::PipelineDescription desc;
desc.addModule({ "source", "FileSource", {} });
desc.addModule({ "sink", "FileSink", {} });
desc.addConnection("source", "sink.input");

std::ofstream out("generated_pipeline.json");
out << desc.toJson();

Key Design Advantages

• Separation of Concerns: JSON parsing logic is isolated in JsonParser, while PipelineDescription remains a pure data structure with no format dependencies.
• Extensible Type System: The PropertyValue variant allows the same IR to support future formats like TOML or YAML without changing the module interfaces.
• Explicit Error Reporting: The ParseResult wrapper provides structured error messages rather than exceptions escaping across API boundaries.
• Round-Trip Fidelity: The toJson output structure matches the parser's input schema, ensuring that saved configurations can be reliably reloaded.

Summary

• ApraPipes uses PipelineDescription as an intermediate representation for pipeline configuration serialization and deserialization, decoupling the JSON format from runtime logic.
• The JsonParser class in base/src/declarative/JsonParser.cpp provides parseFile and parseString methods that convert JSON into C++ structs using section-specific handlers.
• Property values are stored in a std::variant type (PropertyValue) supporting scalars and arrays for flexible configuration.
• The toJson method in base/src/declarative/PipelineDescription.cpp exports the IR back to JSON with proper escaping and formatting.
• The system requires ensureBuiltinModulesRegistered() to validate module types during deserialization and uses ParseResult for error aggregation.

Frequently Asked Questions

What file formats does ApraPipes support for pipeline configuration?

Currently, ApraPipes officially supports JSON for pipeline configuration serialization and deserialization. The intermediate representation (PipelineDescription) uses standard C++ structs and a std::variant-based property system, which means adding TOML or YAML support only requires implementing new parser classes that populate the same data structures without modifying the core engine.

How does ApraPipes handle different property types when deserializing JSON?

The JsonParser uses a helper function called toPropertyValue to convert JSON values into the PropertyValue variant type. This variant can store int64_t, double, bool, std::string, or homogeneous arrays of these types. During deserialization, the parser inspects the JSON value type and selects the appropriate C++ scalar or array representation for storage in the module's property map.

What happens if a module type is not recognized during parsing?

The parser calls ensureBuiltinModulesRegistered() before processing to ensure built-in types are available. If a module's "type" field references an unregistered or unknown type, the validation logic (typically during the module instantiation phase or within parseModulesSection) will fail and populate the ParseResult.error field with a descriptive message indicating the unrecognized type, allowing the application to handle the error gracefully.

Can I modify a pipeline configuration after loading it from JSON?

Yes. Once JsonParser::parseFile or parseString returns a ParseResult, you can mutate the description field directly. The PipelineDescription provides methods like findModule to locate specific modules and modify their properties map, or addConnection to wire new components. After modification, calling toJson() exports the updated configuration to a new JSON document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}