# How to Handle Metadata Merge Conflicts in Multi-Site CMF Deployments

> Learn how CMF automatically resolves metadata merge conflicts in multi-site deployments, preventing errors and ensuring idempotent operations for seamless synchronization.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: how-to-guide
- Published: 2026-03-03

---

**CMF resolves metadata merge conflicts automatically by catching `AlreadyExistsError` exceptions and updating existing entries rather than aborting, while pre-filtering duplicate executions to ensure idempotent push and pull operations across distributed sites.**

In multi-site deployments of the Continuous Metadata Framework (CMF), simultaneous pipeline execution across geographically distributed locations creates inevitable metadata collisions. The [Hewlett Packard Enterprise CMF](https://github.com/hewlettpackard/cmf) repository implements a graceful conflict resolution strategy that transforms potential merge failures into seamless metadata synchronization, ensuring that running `cmf metadata push` repeatedly from multiple sites never corrupts the MLMD store.

## Understanding the Conflict Resolution Architecture

### Conflict Detection via AlreadyExistsError

When the MLMD (ML Metadata) library attempts to insert a **context**, **execution**, or **artifact** that already exists in the target store, it raises `AlreadyExistsError`. In [`cmflib/cmf_merger.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_merger.py), the `handle_context`, `handle_execution`, and `handle_event` functions catch these exceptions to trigger update routines rather than failing the entire operation.

### Pre-Filtering Duplicate Executions

Before any network transfer occurs, the federation layer in [`cmflib/cmf_federation.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_federation.py) minimizes payload size and prevents unnecessary conflicts. The `identify_existing_and_new_executions` function queries the target store using `CmfQuery.get_all_executions_in_pipeline`, computes the intersection with incoming UUIDs, and filters out executions that already exist remotely.

### The Unified Merge API

Both server-side merges and client-side federation operations rely on the same high-level methods defined in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py). The `Cmf` class provides `merge_created_context`, `merge_created_execution`, `update_context`, `update_execution`, and `update_existing_artifact` to ensure consistent conflict handling whether using the CLI or the REST API endpoint `/mlmd_push`.

## Step-by-Step Conflict Resolution Flow

The merge process follows a deterministic pipeline that guarantees **idempotent operations** across all sites:

1. **Pre-filtering**: `identify_existing_and_new_executions` in [`cmflib/cmf_federation.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_federation.py) retrieves all execution UUIDs from the target store and removes duplicates from the payload.

2. **Payload validation**: In `update_mlmd`, if no new executions remain after filtering, the function returns `"exists"` immediately, avoiding unnecessary processing.

3. **Context insertion**: `handle_context` in [`cmflib/cmf_merger.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_merger.py) attempts to create the context. If `AlreadyExistsError` is raised, it falls back to `Cmf.update_context` to merge new `custom_properties` into the existing entry.

4. **Execution insertion**: `handle_execution` follows the same pattern, calling `Cmf.merge_created_execution` initially, then `Cmf.update_execution` on conflict to preserve new custom properties.

5. **Artifact handling**: `handle_event` logs artifacts and invokes `Cmf.update_existing_artifact` when duplicates are detected, ensuring the latest version information is preserved.

6. **Final persistence**: `parse_json_to_mlmd` orchestrates the entire walk through stages, executions, and events, with all conflicts resolved automatically before committing to the MLMD store.

## Practical Implementation Examples

### Pushing Metadata via CLI

The simplest way to merge metadata with automatic conflict resolution is through the command line interface:

```bash

# From a directory containing mlmd.json

cmf metadata push --pipeline-name MyPipeline --filename mlmd.json

```

Under the hood, `CmdMetadataPush.run()` defined in [`cmflib/commands/metadata/push.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/metadata/push.py) builds the request and sends it to the server endpoint in [`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py). The server invokes `cmflib.cmf_federation.update_mlmd()`, which executes the conflict resolution flow described above.

### Pulling Metadata to Downstream Sites

When synchronizing metadata to a local site, conflicts are handled symmetrically:

```bash
cmf metadata pull --pipeline-name MyPipeline --output-dir ./mlmd_pull

```

The client contacts the server's `/mlmd_pull` endpoint. The `update_mlmd` function loads the server store, filters out executions already present locally using the same UUID comparison logic, and calls `parse_json_to_mlmd` to merge only missing metadata into the client's MLMD store.

### Programmatic Merge with Python

For custom automation or integration with existing MLOps pipelines, use the Python API directly:

```python
from cmflib.cmf_federation import update_mlmd
from cmflib.cmfquery import CmfQuery

# Initialize query object pointing at target server store

query = CmfQuery(filepath="/var/lib/cmf_server/mlmd_store")

# Load payload from file or network

with open("mlmd.json") as f:
    payload = f.read()

# Execute push with automatic conflict resolution

status = update_mlmd(
    query, 
    payload,
    pipeline_name="MyPipeline",
    cmd="push",
    exe_uuid=None
)

print("Merge status:", status)  # Returns "success", "exists", or "version_update"

```

This approach gives you full control over the merge process while maintaining the same conflict guarantees as the CLI.

## Key Source Files and Functions

Understanding the repository structure helps when debugging complex multi-site scenarios:

- **[`cmflib/cmf_merger.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_merger.py)**: Contains core merge logic including `handle_context`, `handle_execution`, and `handle_event`. All conflict handling routes through the `AlreadyExistsError` exception handlers in this file.

- **[`cmflib/cmf_federation.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_federation.py)**: Implements the federation layer with `identify_existing_and_new_executions` and `update_mlmd`. This file orchestrates the pre-filtering logic that makes multi-site deployments efficient.

- **[`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py)**: Defines the high-level `Cmf` class methods that perform actual updates: `merge_created_context`, `update_context`, `update_execution`, and `update_existing_artifact`.

- **[`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py)**: Hosts the REST endpoints `/mlmd_push` and `/mlmd_pull` that delegate to federation functions, enabling remote conflict resolution.

## Summary

- **Automatic detection**: CMF catches `AlreadyExistsError` from the MLMD library to identify conflicts rather than failing operations.
- **Smart pre-filtering**: The federation layer removes duplicate executions before network transfer by comparing UUIDs against the target store.
- **Property merging**: On conflict, CMF updates existing entries to merge new `custom_properties` rather than overwriting entire records.
- **Idempotent operations**: Running `cmf metadata push` or `pull` repeatedly from multiple sites never corrupts the store or creates duplicate entries.
- **Unified interface**: Whether using CLI commands, REST APIs, or direct Python calls, all entry points share the same conflict resolution logic in [`cmf_merger.py`](https://github.com/hewlettpackard/cmf/blob/main/cmf_merger.py).

## Frequently Asked Questions

### What happens if two sites push the same execution simultaneously?

CMF resolves this race condition gracefully. The first write succeeds normally, while the second triggers an `AlreadyExistsError` in `handle_execution` within [`cmflib/cmf_merger.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_merger.py). The framework catches this exception and calls `Cmf.update_execution` to merge any new custom properties from the second push into the existing record, ensuring no data loss occurs.

### Does CMF support manual conflict resolution for metadata merges?

No manual intervention is required. According to the hewlettpackard/cmf source code, all conflicts are resolved automatically using the update methods defined in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py). The system assumes that newer custom properties should be merged into existing entries, making the process fully automated for CI/CD pipelines.

### How does CMF prevent duplicate artifacts during multi-site synchronization?

Before merging, `identify_existing_and_new_executions` in [`cmflib/cmf_federation.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_federation.py) computes the intersection of execution UUIDs between the incoming payload and the target store. If an artifact's parent execution already exists, the payload is filtered accordingly. During the final merge in `handle_event`, any remaining artifact conflicts trigger `Cmf.update_existing_artifact`, which preserves the latest version information rather than creating duplicates.

### Can I use the conflict resolution logic outside of the CMF CLI?

Yes. The same merge capabilities are available programmatically by importing `update_mlmd` from `cmflib.cmf_federation` and `CmfQuery` from `cmflib.cmfquery`. This allows custom MLOps tools to leverage CMF's conflict resolution when synchronizing metadata between proprietary systems and CMF-compliant stores.