How Artifact Version Tracking Prevents Duplicate Dataset Logging in CMF

CMF prevents duplicate dataset logging by using content-derived hashes as unique artifact URIs, querying the ML Metadata store for existing artifacts with matching URIs before creating new ones, and linking new executions to existing artifacts when duplicates are detected.

CMF (Collaborative Metadata Framework) from Hewlett Packard Enterprise automatically eliminates redundant dataset records by treating version identifiers as canonical artifact addresses. When you log a dataset, the framework computes a content-based hash that serves as the artifact's unique URI, ensuring identical data files never create duplicate metadata entries.

Content-Addressable Versioning

In cmflib/cmf.py (lines 68‑70), the framework computes a version identifier by calling dvc_get_hash(url) to generate a DVC hash of the file content. This hash becomes the artifact's URI in the ML Metadata store, transforming the dataset's binary content into a unique, deterministic address. If you provide an explicit version via log_dataset_with_version, that value is used instead of computing a fresh hash.

The Duplicate Detection Query

Before creating any metadata record, CMF queries the store for existing collisions. In cmflib/cmf_server.py (lines 30‑33), the system executes:

existing_artifact.extend(self.store.get_artifacts_by_uri(c_hash))

This get_artifacts_by_uri call—provided by cmflib/metadata_helper.py and implemented in the storage layer (cmflib/store/cmfstore.py)—searches for any artifact already registered with that content hash.

Reuse or Create: The Deduplication Logic

The framework branches based on whether the query returns a match (lines 34‑62 in cmflib/cmf_server.py):

When a duplicate is detected, CMF updates the existing artifact's properties and links the current execution rather than creating redundant metadata:

if existing_artifact and len(existing_artifact) != 0:
    existing_artifact = existing_artifact[0]
    self.update_existing_artifact(existing_artifact, custom_props)
    uri = c_hash
    self.update_dataset_url(existing_artifact, props.get("url", ""))
    artifact = link_execution_to_artifact(...)

When no match exists, it generates a new artifact using the hash as the URI, falling back to a UUID only if the hash is empty:

uri = c_hash if c_hash and c_hash.strip() else str(uuid.uuid1())
artifact = create_new_artifact_event_and_attribution(...)

Because the URI is derived from the dataset's content hash, two log calls referring to the same data produce identical URIs. The lookup detects the collision, and the framework links the new execution to the existing artifact rather than inserting a second record.

Practical Implementation Examples

Explicit version logging via log_dataset_with_version allows you to supply a pre-computed hash:

from cmflib.cmf import Cmf

cmf = Cmf()
artifact = cmf.log_dataset_with_version(
    url="data/train.csv",
    version="a1b2c3d4",
    event="input",
    props={"url": "data/train.csv"},
)

Automatic hash computation via log_dataset handles versioning transparently:

artifact = cmf.log_dataset(
    url="data/train.csv",
    event="input",
    custom_properties={"owner": "alice"},
)

Both methods route through the same deduplication logic in cmflib/cmf_server.py, ensuring that repeated logging of identical data only creates new execution-to-artifact links without duplicating the underlying dataset record.

Summary

• Content hashes serve as URIs: CMF uses DVC hashes (or caller-supplied versions) as the canonical URI for each artifact, making dataset identity deterministic based on file content.
• Lookup before creation: The system queries get_artifacts_by_uri to detect existing artifacts before instantiating new ones.
• Execution linking: When duplicates are detected, CMF updates the existing artifact's metadata and creates a new execution-to-artifact link, avoiding redundant records.
• UUID fallback: If no hash is available, the framework generates a UUID to ensure the artifact still has a unique identifier.

Frequently Asked Questions

What happens if I log the same dataset twice in CMF?

The framework detects the existing artifact by its content hash (URI) and links the new execution to that existing record instead of creating a duplicate. It updates the artifact's properties and dataset URL if provided, but no second artifact entry is created in the ML Metadata store.

How does CMF compute the version identifier for duplicate detection?

CMF uses dvc_get_hash(url) to generate a DVC hash of the file content, which becomes the artifact's URI in cmflib/cmf.py. This content-derived hash ensures that identical datasets produce identical identifiers regardless of when or where they are logged.

Can I bypass automatic hash generation and provide my own version?

Yes, by calling log_dataset_with_version and passing a custom version parameter, you supply the URI value directly. The system still queries get_artifacts_by_uri to check for existing artifacts with that version before deciding whether to create a new artifact or reuse the existing one.

Which source files contain the core deduplication logic?

The primary implementation resides in cmflib/cmf_server.py (lines 30‑62), where get_artifacts_by_uri performs the lookup and the conditional logic decides between updating existing artifacts and creating new ones. The hash computation originates in cmflib/cmf.py, while the storage interface is defined in cmflib/metadata_helper.py and implemented in cmflib/store/cmfstore.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}