How Execution Reuse Works with `create_new_execution` in CMF

Set create_new_execution=False in Cmf.create_execution() to reuse existing MLMD executions by name, or use the default True to always create fresh executions.

The Hewlett Packard Enterprise CMF (Context Management Framework) library enables fine-grained control over ML Metadata (MLMD) execution creation through the create_new_execution parameter. When building reproducible machine learning pipelines with the hewlettpackard/cmf repository, understanding how this boolean flag governs execution reuse is essential for managing pipeline state and lineage across runs.

Understanding the create_new_execution Flag

The create_new_execution parameter is a boolean flag that defaults to True in Cmf.create_execution(). This flag determines whether the library treats the execution_type argument as a generic type identifier or as a persistent name for lookup purposes. When set to False, the framework enables deterministic execution reuse by storing the type value as a custom name property, allowing subsequent calls to retrieve and update the same execution record.

The Execution Creation Flow

The implementation spans two core files in cmflib. In cmflib/cmf.py, the create_execution() method (lines 61-102) receives the flag and delegates to create_new_execution_in_existing_run_context() (lines 384-398). This delegation passes control to cmflib/metadata_helper.py, where create_execution_with_type() (lines 66-90) implements the branching logic:

• create_new_execution=True (default): The function always instantiates a brand-new ml_metadata.proto.Execution of the specified type. The execution carries no name property, making it impossible to retrieve later for reuse.

• create_new_execution=False: The function first queries existing executions for a matching type_name and name combination. If found, it returns the existing execution and updates its properties. If no match exists, it creates a new execution with the name property explicitly set to the execution_type value.

Code Examples

Default Behavior: Always Create Fresh Executions

When you omit the flag or pass True, every call generates a distinct execution without storage names.

from cmflib.cmf import Cmf

cmf = Cmf(filepath="mlmd", pipeline_name="my_pipeline")
cmf.create_context(pipeline_stage="train")

# Each call creates a distinct execution

exec_1 = cmf.create_execution(
    execution_type="TrainingStep",
    custom_properties={"epoch": 1}
)  # Execution ID: 101

exec_2 = cmf.create_execution(
    execution_type="TrainingStep",
    custom_properties={"epoch": 2}
)  # Execution ID: 102 (new, not reused)

Reuse Mode: Linking to Existing Executions

Set the flag to False to enable reuse. The execution_type becomes the execution's persistent name.

# First call creates a named execution

exec_reuse = cmf.create_execution(
    execution_type="Preprocessing",
    custom_properties={"dataset_version": "v1.0"},
    create_new_execution=False
)  # Execution ID: 201, name="Preprocessing"

# Second call retrieves and updates the same execution

exec_same = cmf.create_execution(
    execution_type="Preprocessing",
    custom_properties={"dataset_version": "v1.1"},
    create_new_execution=False
)  # Execution ID: 201 (reused), properties updated

Mixed Mode Limitations

Executions created with the default behavior can never be reused later, even if you subsequently pass create_new_execution=False.

# Create execution without reuse capability

exec_no_name = cmf.create_execution("Evaluation")  # ID: 301

# Attempting to reuse fails because the original lacks a name property

exec_attempt = cmf.create_execution(
    execution_type="Evaluation",
    create_new_execution=False
)  # ID: 302 (new execution created, not reused)

Summary

• The create_new_execution parameter in Cmf.create_execution() controls whether CMF creates fresh MLMD executions or reuses existing ones by name.
• Default True: Always creates new executions without storing a name property, preventing future reuse.
• False mode: Stores execution_type as the execution's name custom property, enabling lookup and reuse across pipeline runs.
• Reuse only works for executions originally created with create_new_execution=False according to the hewlettpackard/cmf source code in cmflib/metadata_helper.py.
• Core logic resides in the create_execution_with_type() function, which queries existing executions by type and name when the flag is disabled.

Frequently Asked Questions

Can I reuse an execution that was created with the default create_new_execution=True?

No. Executions created with create_new_execution=True (the default) do not store the name custom property in MLMD. Since the reuse lookup in cmflib/metadata_helper.py specifically queries for executions matching both type and name, these executions remain invisible to the reuse mechanism. You must create the original execution with create_new_execution=False to enable future reuse.

Where does CMF store the execution name for reuse lookups?

CMF stores the execution name as a custom property within the MLMD Execution proto. When create_new_execution=False, the create_execution_with_type() function in cmflib/metadata_helper.py explicitly sets the name property to the value of execution_type. This property serves as the lookup key when subsequent calls attempt to reuse the execution.

What happens if I pass different custom_properties when reusing an execution?

The existing execution's properties are updated with the new values provided in the custom_properties dictionary. The function returns the same execution ID but modifies the execution's metadata to reflect the latest property values passed during the reuse call, as implemented in cmflib/metadata_helper.py lines 80-90.

Is it safe to use create_new_execution=False in concurrent pipeline runs?

Concurrent runs accessing the same MLMD backend with identical execution names and types will compete for the same execution record. The underlying MLMD store handles atomicity, but concurrent updates to custom properties may result in last-write-wins behavior. For isolated parallel runs, use distinct execution names or pipeline contexts rather than relying on reuse across concurrent processes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}