# How Artifact Version Tracking Prevents Duplicate Dataset Logging in CMF

> Learn how CMF artifact version tracking prevents duplicate dataset logging using content-derived hashes and ML Metadata to ensure data integrity. Avoid redundant logging effectively.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: how-to-guide
- Published: 2026-03-03

---

**CMF prevents duplicate dataset logging by using content-derived hashes as unique artifact URIs, querying the ML Metadata store for existing artifacts with matching URIs before creating new ones, and linking new executions to existing artifacts when duplicates are detected.**

CMF (Collaborative Metadata Framework) from Hewlett Packard Enterprise automatically eliminates redundant dataset records by treating version identifiers as canonical artifact addresses. When you log a dataset, the framework computes a content-based hash that serves as the artifact's unique URI, ensuring identical data files never create duplicate metadata entries.

## Content-Addressable Versioning

In [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) (lines 68‑70), the framework computes a **version identifier** by calling `dvc_get_hash(url)` to generate a DVC hash of the file content. This hash becomes the artifact's **URI** in the ML Metadata store, transforming the dataset's binary content into a unique, deterministic address. If you provide an explicit version via `log_dataset_with_version`, that value is used instead of computing a fresh hash.

## The Duplicate Detection Query

Before creating any metadata record, CMF queries the store for existing collisions. In [`cmflib/cmf_server.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_server.py) (lines 30‑33), the system executes:

```python
existing_artifact.extend(self.store.get_artifacts_by_uri(c_hash))

```

This `get_artifacts_by_uri` call—provided by [`cmflib/metadata_helper.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/metadata_helper.py) and implemented in the storage layer ([`cmflib/store/cmfstore.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/store/cmfstore.py))—searches for any artifact already registered with that content hash.

## Reuse or Create: The Deduplication Logic

The framework branches based on whether the query returns a match (lines 34‑62 in [`cmflib/cmf_server.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_server.py)):

**When a duplicate is detected**, CMF updates the existing artifact's properties and links the current execution rather than creating redundant metadata:

```python
if existing_artifact and len(existing_artifact) != 0:
    existing_artifact = existing_artifact[0]
    self.update_existing_artifact(existing_artifact, custom_props)
    uri = c_hash
    self.update_dataset_url(existing_artifact, props.get("url", ""))
    artifact = link_execution_to_artifact(...)

```

**When no match exists**, it generates a new artifact using the hash as the URI, falling back to a UUID only if the hash is empty:

```python
uri = c_hash if c_hash and c_hash.strip() else str(uuid.uuid1())
artifact = create_new_artifact_event_and_attribution(...)

```

Because the URI is derived from the dataset's content hash, two log calls referring to the same data produce identical URIs. The lookup detects the collision, and the framework links the new execution to the existing artifact rather than inserting a second record.

## Practical Implementation Examples

**Explicit version logging** via `log_dataset_with_version` allows you to supply a pre-computed hash:

```python
from cmflib.cmf import Cmf

cmf = Cmf()
artifact = cmf.log_dataset_with_version(
    url="data/train.csv",
    version="a1b2c3d4",
    event="input",
    props={"url": "data/train.csv"},
)

```

**Automatic hash computation** via `log_dataset` handles versioning transparently:

```python
artifact = cmf.log_dataset(
    url="data/train.csv",
    event="input",
    custom_properties={"owner": "alice"},
)

```

Both methods route through the same deduplication logic in [`cmflib/cmf_server.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_server.py), ensuring that repeated logging of identical data only creates new execution-to-artifact links without duplicating the underlying dataset record.

## Summary

- **Content hashes serve as URIs**: CMF uses DVC hashes (or caller-supplied versions) as the canonical URI for each artifact, making dataset identity deterministic based on file content.
- **Lookup before creation**: The system queries `get_artifacts_by_uri` to detect existing artifacts before instantiating new ones.
- **Execution linking**: When duplicates are detected, CMF updates the existing artifact's metadata and creates a new execution-to-artifact link, avoiding redundant records.
- **UUID fallback**: If no hash is available, the framework generates a UUID to ensure the artifact still has a unique identifier.

## Frequently Asked Questions

### What happens if I log the same dataset twice in CMF?

The framework detects the existing artifact by its content hash (URI) and links the new execution to that existing record instead of creating a duplicate. It updates the artifact's properties and dataset URL if provided, but no second artifact entry is created in the ML Metadata store.

### How does CMF compute the version identifier for duplicate detection?

CMF uses `dvc_get_hash(url)` to generate a DVC hash of the file content, which becomes the artifact's URI in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py). This content-derived hash ensures that identical datasets produce identical identifiers regardless of when or where they are logged.

### Can I bypass automatic hash generation and provide my own version?

Yes, by calling `log_dataset_with_version` and passing a custom `version` parameter, you supply the URI value directly. The system still queries `get_artifacts_by_uri` to check for existing artifacts with that version before deciding whether to create a new artifact or reuse the existing one.

### Which source files contain the core deduplication logic?

The primary implementation resides in [`cmflib/cmf_server.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf_server.py) (lines 30‑62), where `get_artifacts_by_uri` performs the lookup and the conditional logic decides between updating existing artifacts and creating new ones. The hash computation originates in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py), while the storage interface is defined in [`cmflib/metadata_helper.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/metadata_helper.py) and implemented in [`cmflib/store/cmfstore.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/store/cmfstore.py).