# Pipeline Configuration Serialization and Deserialization in ApraPipes

> Learn how ApraPipes manages pipeline configuration serialization and deserialization using an intermediate PipelineDescription representation for seamless JSON to C++ data structure conversion.

- Repository: [Apra Labs/aprapipes](https://github.com/apra-labs/aprapipes)
- Tags: internals
- Published: 2026-02-25

---

**ApraPipes handles pipeline configuration serialization and deserialization through an intermediate representation (IR) called `PipelineDescription`, enabling round-trip conversion between JSON documents and C++ data structures via the `JsonParser` class and the `toJson` method.**

The apra-labs/aprapipes repository implements a type-safe system for pipeline configuration serialization and deserialization that decouples the persistence format from the runtime engine. By representing pipelines as a hierarchy of plain C++ structs, the framework supports JSON as a transport and debugging format while maintaining strict in-memory data models for modules, connections, and global settings.

## The Intermediate Representation (IR) Architecture

ApraPipes defines its pipeline IR through a set of simple structs that aggregate configuration data without any JSON-specific logic.

### Core Data Structures

The `PipelineDescription` struct acts as the root container and is composed of three primary elements:

- **PipelineSettings**: Stores global options including `name`, `version`, `description`, `queue_size`, `on_error`, and `auto_start`.
- **ModuleInstance**: Represents a single module with `instance_id`, `module_type`, and a `properties` map of type `std::map<std::string, PropertyValue>`.
- **Connection**: Defines wires between pins using `from_module`, `from_pin`, `to_module`, `to_pin`, and an optional `sieve` flag.

The `PropertyValue` type is implemented as a `std::variant` capable of holding scalar types (`int64_t`, `double`, `bool`, `std::string`) or homogeneous arrays of those scalars. This design allows the same IR to support multiple configuration formats without modification.

## Deserializing Pipeline Configurations from JSON

The `JsonParser` class in [`base/src/declarative/JsonParser.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/JsonParser.cpp) implements the complete deserialization pipeline, converting JSON inputs into populated `PipelineDescription` objects.

### Entry Points and Initial Processing

Deserialization begins through two static entry points:

- `JsonParser::parseFile(const std::string& path)` loads JSON from disk using the *nlohmann/json* library.
- `JsonParser::parseString(const std::string& json, const std::string& sourceName)` parses inline JSON strings.

Both methods forward the root JSON object to a shared `parse` helper that returns a `ParseResult` struct containing the populated `description` and an optional `error` string.

### Section-by-Section Parsing Logic

The parser delegates processing to four specialized static functions that walk specific JSON sections:

1. **`parsePipelineSection`**: Extracts top-level metadata fields such as `name`, `version`, and `description` from the `pipeline` object.
2. **`parseSettingsSection`**: Reads global configuration under `settings`, supporting both camelCase (`queueSize`) and snake_case (`queue_size`) keys for parameters like error handling and auto-start behavior.
3. **`parseModulesSection`**: Iterates over the `modules` object, creating a `ModuleInstance` for each key. It extracts the mandatory `"type"` field and optional `"props"` map, converting each property value to `PropertyValue` via the `toPropertyValue` helper.
4. **`parseConnectionsSection`**: Validates the `connections` array, extracts `"from"` and `"to"` strings, and parses the `module.pin` syntax using `Connection::parse`. It optionally reads a `sieve` boolean flag.

### Module Registration and Error Handling

Before parsing, the framework calls `ensureBuiltinModulesRegistered()` to guarantee that built-in module types are available for validation. Throughout the process, errors throw `std::runtime_error` with descriptive messages that are caught and aggregated into the `ParseResult.error` field, allowing the caller to distinguish between success and failure states without crashing.

## Serializing Pipeline Configurations to JSON

The `PipelineDescription::toJson` method in [`base/src/declarative/PipelineDescription.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/PipelineDescription.cpp) handles serialization, manually constructing a JSON string that mirrors the format accepted by the parser.

### Struct-to-JSON Conversion

The serialization process walks the IR and emits formatted output:

- **Settings Block**: Writes the `PipelineSettings` struct using `escapeJson` for string safety.
- **Source Block**: Emits optional `source_format` and `source_path` fields.
- **Modules Array**: For each `ModuleInstance`, serializes `instance_id`, `module_type`, and the `properties` map using `propertyValueToString`. This helper formats scalars, quoted strings, and array literals (`[ ... ]`) appropriately.
- **Connections Array**: Serializes each `Connection` by concatenating `module.pin` strings, omitting the pin suffix if empty.

The resulting string is a valid JSON document suitable for debugging, network transport, or persistence to disk.

## Practical Workflow Examples

### Loading and Modifying a Pipeline

The following example demonstrates loading a configuration, mutating a property, and re-serializing:

```cpp
#include "declarative/JsonParser.h"
#include "declarative/PipelineDescription.h"

int main() {
    // Deserialize from file
    auto result = apra::JsonParser::parseFile("examples/basic/split_pipeline.json");
    if (!result.success) {
        std::cerr << "Parse error: " << result.error << '\n';
        return 1;
    }

    // Modify the in-memory IR
    auto* decoder = result.description.findModule("my_decoder");
    if (decoder) {
        decoder->properties["bitrate"] = int64_t(4000000);
    }

    // Serialize back to JSON
    std::string json = result.description.toJson();
    std::cout << json << std::endl;
}

```

### Parsing Inline Configuration

For embedded configurations, use `parseString`:

```cpp
std::string json = R"({
  "pipeline": { "name": "demo", "version": "1.0" },
  "settings": { "queueSize": 5, "autoStart": true },
  "modules": {
    "source": { "type": "FileSource", "props": { "path": "video.mp4" } }
  },
  "connections": [{ "from": "source", "to": "decoder" }]
})";

auto result = apra::JsonParser::parseString(json, "<inline>");
if (result.success) {
    std::cout << "Loaded: " << result.description.settings.name << '\n';
}

```

### Programmatic Construction and Export

You can also build pipelines in code and export them:

```cpp
apra::PipelineDescription desc;
desc.addModule({ "source", "FileSource", {} });
desc.addModule({ "sink", "FileSink", {} });
desc.addConnection("source", "sink.input");

std::ofstream out("generated_pipeline.json");
out << desc.toJson();

```

## Key Design Advantages

- **Separation of Concerns**: JSON parsing logic is isolated in `JsonParser`, while `PipelineDescription` remains a pure data structure with no format dependencies.
- **Extensible Type System**: The `PropertyValue` variant allows the same IR to support future formats like TOML or YAML without changing the module interfaces.
- **Explicit Error Reporting**: The `ParseResult` wrapper provides structured error messages rather than exceptions escaping across API boundaries.
- **Round-Trip Fidelity**: The `toJson` output structure matches the parser's input schema, ensuring that saved configurations can be reliably reloaded.

## Summary

- ApraPipes uses `PipelineDescription` as an intermediate representation for pipeline configuration serialization and deserialization, decoupling the JSON format from runtime logic.
- The `JsonParser` class in [`base/src/declarative/JsonParser.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/JsonParser.cpp) provides `parseFile` and `parseString` methods that convert JSON into C++ structs using section-specific handlers.
- Property values are stored in a `std::variant` type (`PropertyValue`) supporting scalars and arrays for flexible configuration.
- The `toJson` method in [`base/src/declarative/PipelineDescription.cpp`](https://github.com/apra-labs/aprapipes/blob/main/base/src/declarative/PipelineDescription.cpp) exports the IR back to JSON with proper escaping and formatting.
- The system requires `ensureBuiltinModulesRegistered()` to validate module types during deserialization and uses `ParseResult` for error aggregation.

## Frequently Asked Questions

### What file formats does ApraPipes support for pipeline configuration?

Currently, ApraPipes officially supports JSON for pipeline configuration serialization and deserialization. The intermediate representation (`PipelineDescription`) uses standard C++ structs and a `std::variant`-based property system, which means adding TOML or YAML support only requires implementing new parser classes that populate the same data structures without modifying the core engine.

### How does ApraPipes handle different property types when deserializing JSON?

The `JsonParser` uses a helper function called `toPropertyValue` to convert JSON values into the `PropertyValue` variant type. This variant can store `int64_t`, `double`, `bool`, `std::string`, or homogeneous arrays of these types. During deserialization, the parser inspects the JSON value type and selects the appropriate C++ scalar or array representation for storage in the module's property map.

### What happens if a module type is not recognized during parsing?

The parser calls `ensureBuiltinModulesRegistered()` before processing to ensure built-in types are available. If a module's `"type"` field references an unregistered or unknown type, the validation logic (typically during the module instantiation phase or within `parseModulesSection`) will fail and populate the `ParseResult.error` field with a descriptive message indicating the unrecognized type, allowing the application to handle the error gracefully.

### Can I modify a pipeline configuration after loading it from JSON?

Yes. Once `JsonParser::parseFile` or `parseString` returns a `ParseResult`, you can mutate the `description` field directly. The `PipelineDescription` provides methods like `findModule` to locate specific modules and modify their `properties` map, or `addConnection` to wire new components. After modification, calling `toJson()` exports the updated configuration to a new JSON document.