How to Use CMF CLI Commands for Metadata Push/Pull Operations

The CMF (Common Metadata Framework) CLI provides cmf metadata push and cmf metadata pull commands to synchronize ML metadata between local workstations and a CMF server, enabling seamless upload of execution records and download of remote pipeline states.

The Hewlett Packard Enterprise CMF repository (hewlettpackard/cmf) offers a production-ready command-line interface for managing machine learning metadata across distributed teams. When you use CMF CLI commands for metadata push/pull operations, you transfer mlmd (ML Metadata) files between your local environment and a centralized CMF server to maintain consistency in pipeline tracking, artifact lineage, and experiment reproducibility.

Command Architecture and Entry Points

The metadata commands are registered in cmflib/cli/parser.py as sub-commands of the metadata group. Each command inherits from the abstract base class CmdBase defined in cmflib/cli/command.py. When you execute cmf metadata push or cmf metadata pull, the parser instantiates either CmdMetadataPush or CmdMetadataPull and invokes their respective run() methods.

Both commands begin with identical initialization logic to establish server connectivity. They call fetch_cmf_config_path() to locate the .cmfconfig file and extract the server URL via CmfConfig.read_config():

output, cmf_config_path = fetch_cmf_config_path()
attr_dict = CmfConfig.read_config(cmf_config_path)
url = attr_dict.get("cmf-server-url", "http://127.0.0.1:80")

Pushing Local Metadata to the Server

The metadata push command uploads your local MLMD database and optional artifacts to the CMF server. According to the hewlettpackard/cmf source code, the implementation resides in cmflib/commands/metadata/push.py.

Workflow and Implementation

When you run cmf metadata push, the command executes the following sequence:

1. Locate the local MLMD file – Defaults to ./mlmd if you omit the -f/--file_name argument.
2. Load and serialize – Uses CmfQuery(mlmd_file_name) to load the database, then converts it to JSON via query.dumptojson(pipeline_name, None).
3. Upload metadata – Posts the JSON payload to /api/mlmd_push using server_interface.call_mlmd_push().
4. Handle responses – Interprets server responses to detect version mismatches (UpdateCmfVersion), duplicate executions, or missing pipelines (PipelineNotFound).
5. Upload supplementary artifacts – Automatically discovers and uploads Python environment files and artifact labels via call_python_env() and call_label().

Uploading TensorBoard Logs

If you provide the -t/--tensorboard_path flag, the command invokes server_interface.call_tensorboard() to upload either a single file or an entire directory of TensorBoard logs to the server.

Push Command Examples

Push the default ./mlmd file for a specific pipeline:

cmf metadata push -p my_pipeline

Push a specific file location and include TensorBoard logs:

cmf metadata push -p my_pipeline -f path/to/mlmd -t /tmp/tensorboard_logs

Push metadata for a specific execution UUID:

cmf metadata push -p my_pipeline -e f9da581c-d16c-11ef-9809-9350156ed1ac

Successful execution returns a MlmdFilePushSuccess response and prints confirmation:

metadata push started
........................................
mlmd is successfully pushed.

Pulling Remote Metadata to Local Storage

The metadata pull command downloads ML metadata from the CMF server and merges it into your local database. The implementation is located in cmflib/commands/metadata/pull.py.

Workflow and Implementation

The pull operation follows this server-client synchronization pattern:

1. Determine output location – Defaults to ./mlmd if -f/--file_name is not specified.
2. Request metadata – Sends a POST request to /api/mlmd_pull via server_interface.call_mlmd_pull(), passing the pipeline name and optional execution UUID.
3. Error handling – Interprets HTTP 404 as PipelineNotFound and specific payload strings indicating ExecutionUUIDNotFound.
4. Local merge – Uses update_mlmd() from cmflib/cmf_federation.py to parse the received JSON and create or update the local MLMD SQLite store, respecting version-checking logic to prevent conflicts.
5. Return status – Returns MlmdFilePullSuccess on completion or raises UpdateCmfVersion/MlmdNotFoundOnServer for error conditions.

Pull Command Examples

Pull the latest metadata for a pipeline:

cmf metadata pull -p my_pipeline

Pull to a custom file location:

cmf metadata pull -p my_pipeline -f ./downloaded_mlmd

Pull a specific execution UUID:

cmf metadata pull -p my_pipeline -e f9da581c-d16c-11ef-9809-9350156ed1ac

On success, the CLI prints:

mlmd file successfully pulled to ./mlmd

Command-Line Options Reference

The following flags control metadata push and pull operations:

• -p, --pipeline_name – Target pipeline name (required).
• -f, --file_name – Path to local MLMD file; acts as source for push or destination for pull (optional, defaults to ./mlmd).
• -e, --execution_uuid – Specific execution UUID to push or pull (optional).
• -t, --tensorboard_path – Path to TensorBoard logs (push only, optional).

All commands validate inputs using specialized exceptions including MissingArgument, DuplicateArgumentNotAllowed, and PipelineNotFound, which inherit from the base CmfResponse class handled by cmflib/cli/__init__.py.

Summary

• The CMF CLI provides cmf metadata push and cmf metadata pull commands in cmflib/commands/metadata/ to synchronize ML metadata with a centralized server.
• Both commands require a valid .cmfconfig file containing the cmf-server-url and use fetch_cmf_config_path() to locate configuration.
• Push operations serialize local MLMD data using CmfQuery.dumptojson() and upload via call_mlmd_push(), with optional TensorBoard log transfer via call_tensorboard().
• Pull operations download metadata via call_mlmd_pull() and merge locally using update_mlmd() from cmflib/cmf_federation.py.
• The commands handle versioning conflicts through UpdateCmfVersion exceptions and validate pipeline existence with PipelineNotFound.

Frequently Asked Questions

What file format does CMF use for local metadata storage?

CMF uses the MLMD (ML Metadata) format, typically stored as a SQLite database file named mlmd in your working directory. The push and pull commands treat this file as the source of truth for pipeline execution records, artifacts, and lineage information, as implemented in the server interface handlers.

How does CMF handle version conflicts during push or pull operations?

When the server detects a version mismatch between the client and server CMF libraries, it raises an UpdateCmfVersion exception that propagates through the CmfResponse error handling system in cmflib/cli/__init__.py. You must upgrade your local CMF installation to match the server version before retrying the operation.

Can I push metadata for a specific pipeline execution rather than the entire pipeline?

Yes. Both commands accept the -e/--execution_uuid flag to target a specific execution UUID. During push, this filters the metadata sent to the server via the execution_uuid parameter in call_mlmd_push(); during pull, it retrieves only the records associated with that specific execution from the server's database.

What happens if the local MLMD file does not exist when pulling metadata?

If you specify a destination path with -f that does not exist, the update_mlmd() function in cmflib/cmf_federation.py creates a new SQLite database file at that location and populates it with the downloaded metadata. If you use the default ./mlmd path and no file exists, it initializes a new database automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}