How to Handle Metadata Merge Conflicts in Multi-Site CMF Deployments

CMF resolves metadata merge conflicts automatically by catching AlreadyExistsError exceptions and updating existing entries rather than aborting, while pre-filtering duplicate executions to ensure idempotent push and pull operations across distributed sites.

In multi-site deployments of the Continuous Metadata Framework (CMF), simultaneous pipeline execution across geographically distributed locations creates inevitable metadata collisions. The Hewlett Packard Enterprise CMF repository implements a graceful conflict resolution strategy that transforms potential merge failures into seamless metadata synchronization, ensuring that running cmf metadata push repeatedly from multiple sites never corrupts the MLMD store.

Understanding the Conflict Resolution Architecture

Conflict Detection via AlreadyExistsError

When the MLMD (ML Metadata) library attempts to insert a context, execution, or artifact that already exists in the target store, it raises AlreadyExistsError. In cmflib/cmf_merger.py, the handle_context, handle_execution, and handle_event functions catch these exceptions to trigger update routines rather than failing the entire operation.

Pre-Filtering Duplicate Executions

Before any network transfer occurs, the federation layer in cmflib/cmf_federation.py minimizes payload size and prevents unnecessary conflicts. The identify_existing_and_new_executions function queries the target store using CmfQuery.get_all_executions_in_pipeline, computes the intersection with incoming UUIDs, and filters out executions that already exist remotely.

The Unified Merge API

Both server-side merges and client-side federation operations rely on the same high-level methods defined in cmflib/cmf.py. The Cmf class provides merge_created_context, merge_created_execution, update_context, update_execution, and update_existing_artifact to ensure consistent conflict handling whether using the CLI or the REST API endpoint /mlmd_push.

Step-by-Step Conflict Resolution Flow

The merge process follows a deterministic pipeline that guarantees idempotent operations across all sites:

1. Pre-filtering: identify_existing_and_new_executions in cmflib/cmf_federation.py retrieves all execution UUIDs from the target store and removes duplicates from the payload.

2. Payload validation: In update_mlmd, if no new executions remain after filtering, the function returns "exists" immediately, avoiding unnecessary processing.

3. Context insertion: handle_context in cmflib/cmf_merger.py attempts to create the context. If AlreadyExistsError is raised, it falls back to Cmf.update_context to merge new custom_properties into the existing entry.

4. Execution insertion: handle_execution follows the same pattern, calling Cmf.merge_created_execution initially, then Cmf.update_execution on conflict to preserve new custom properties.

5. Artifact handling: handle_event logs artifacts and invokes Cmf.update_existing_artifact when duplicates are detected, ensuring the latest version information is preserved.

6. Final persistence: parse_json_to_mlmd orchestrates the entire walk through stages, executions, and events, with all conflicts resolved automatically before committing to the MLMD store.

Practical Implementation Examples

Pushing Metadata via CLI

The simplest way to merge metadata with automatic conflict resolution is through the command line interface:

# From a directory containing mlmd.json

cmf metadata push --pipeline-name MyPipeline --filename mlmd.json

Under the hood, CmdMetadataPush.run() defined in cmflib/commands/metadata/push.py builds the request and sends it to the server endpoint in server/app/main.py. The server invokes cmflib.cmf_federation.update_mlmd(), which executes the conflict resolution flow described above.

Pulling Metadata to Downstream Sites

When synchronizing metadata to a local site, conflicts are handled symmetrically:

cmf metadata pull --pipeline-name MyPipeline --output-dir ./mlmd_pull

The client contacts the server's /mlmd_pull endpoint. The update_mlmd function loads the server store, filters out executions already present locally using the same UUID comparison logic, and calls parse_json_to_mlmd to merge only missing metadata into the client's MLMD store.

Programmatic Merge with Python

For custom automation or integration with existing MLOps pipelines, use the Python API directly:

from cmflib.cmf_federation import update_mlmd
from cmflib.cmfquery import CmfQuery

# Initialize query object pointing at target server store

query = CmfQuery(filepath="/var/lib/cmf_server/mlmd_store")

# Load payload from file or network

with open("mlmd.json") as f:
    payload = f.read()

# Execute push with automatic conflict resolution

status = update_mlmd(
    query, 
    payload,
    pipeline_name="MyPipeline",
    cmd="push",
    exe_uuid=None
)

print("Merge status:", status)  # Returns "success", "exists", or "version_update"

This approach gives you full control over the merge process while maintaining the same conflict guarantees as the CLI.

Key Source Files and Functions

Understanding the repository structure helps when debugging complex multi-site scenarios:

• cmflib/cmf_merger.py: Contains core merge logic including handle_context, handle_execution, and handle_event. All conflict handling routes through the AlreadyExistsError exception handlers in this file.

• cmflib/cmf_federation.py: Implements the federation layer with identify_existing_and_new_executions and update_mlmd. This file orchestrates the pre-filtering logic that makes multi-site deployments efficient.

• cmflib/cmf.py: Defines the high-level Cmf class methods that perform actual updates: merge_created_context, update_context, update_execution, and update_existing_artifact.

• server/app/main.py: Hosts the REST endpoints /mlmd_push and /mlmd_pull that delegate to federation functions, enabling remote conflict resolution.

Summary

• Automatic detection: CMF catches AlreadyExistsError from the MLMD library to identify conflicts rather than failing operations.
• Smart pre-filtering: The federation layer removes duplicate executions before network transfer by comparing UUIDs against the target store.
• Property merging: On conflict, CMF updates existing entries to merge new custom_properties rather than overwriting entire records.
• Idempotent operations: Running cmf metadata push or pull repeatedly from multiple sites never corrupts the store or creates duplicate entries.
• Unified interface: Whether using CLI commands, REST APIs, or direct Python calls, all entry points share the same conflict resolution logic in cmf_merger.py.

Frequently Asked Questions

What happens if two sites push the same execution simultaneously?

CMF resolves this race condition gracefully. The first write succeeds normally, while the second triggers an AlreadyExistsError in handle_execution within cmflib/cmf_merger.py. The framework catches this exception and calls Cmf.update_execution to merge any new custom properties from the second push into the existing record, ensuring no data loss occurs.

Does CMF support manual conflict resolution for metadata merges?

No manual intervention is required. According to the hewlettpackard/cmf source code, all conflicts are resolved automatically using the update methods defined in cmflib/cmf.py. The system assumes that newer custom properties should be merged into existing entries, making the process fully automated for CI/CD pipelines.

How does CMF prevent duplicate artifacts during multi-site synchronization?

Before merging, identify_existing_and_new_executions in cmflib/cmf_federation.py computes the intersection of execution UUIDs between the incoming payload and the target store. If an artifact's parent execution already exists, the payload is filtered accordingly. During the final merge in handle_event, any remaining artifact conflicts trigger Cmf.update_existing_artifact, which preserves the latest version information rather than creating duplicates.

Can I use the conflict resolution logic outside of the CMF CLI?

Yes. The same merge capabilities are available programmatically by importing update_mlmd from cmflib.cmf_federation and CmfQuery from cmflib.cmfquery. This allows custom MLOps tools to leverage CMF's conflict resolution when synchronizing metadata between proprietary systems and CMF-compliant stores.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}