How to Export CMF Lineage Data for External Visualization Tools

You can export CMF lineage data using the cmf metadata export CLI command to generate JSON files, or query the REST API endpoints that return D3-compatible graph structures ready for visualization tools like Neo4j, GraphViz, or custom D3 applications.

The Common Metadata Framework (CMF) captures complete artifact and execution provenance in an ML Metadata (MLMD) store. While CMF provides a built-in UI for lineage visualization, data science teams often need to export this lineage data to external visualization tools for custom dashboards or graph analysis. This guide covers the two primary export pathways—CLI serialization and REST API consumption—using the actual implementation in the hewlettpackard/cmf repository.

Understanding CMF Lineage Export Architecture

The MLMD Store Foundation

CMF persists all pipeline metadata—including artifacts, executions, and contexts—inside an MLMD store. This provenance data powers the four visualization modes documented in docs/ui/lineage.md: Artifact Tree, Execution Tree, Artifact-Execution Tree, and Force-directed layouts. Both export methods ultimately query this same underlying store, but package the results differently for external consumption.

Export Pathways Overview

The repository provides two authoritative methods for accessing lineage data externally:

1. CLI Export: Serialize the entire pipeline metadata to a self-contained JSON file using cmf metadata export (implemented in cmflib/commands/metadata/export.py).
2. REST API: Query purpose-built endpoints that return pre-formatted D3-compatible graphs ( routed in server/app/main.py).

Both approaches leverage the same query helpers—query_execution_lineage_d3tree.py and query_artifact_lineage_d3tree.py—to construct the lineage structures.

Exporting Lineage via the CMF CLI

The CmdMetadataExport Implementation

The command-line interface provides the most straightforward method for bulk exporting lineage data. The CmdMetadataExport class in cmflib/commands/metadata/export.py handles validation, MLMD store connection, and JSON serialization.

When you invoke the export command, the run() method (lines 51–69) instantiates a cmfquery.CmfQuery object, then calls query.dumptojson(pipeline_name, None) (lines 21–23) to extract the complete metadata graph. The output includes artifacts, executions, contexts, and their relationships in a machine-readable format.

Step-by-Step CLI Export Workflow

Execute the following to dump a pipeline's metadata:

# Export the pipeline "my-pipeline" to my-pipeline.json

cmf metadata export -p my-pipeline -j my-pipeline.json

The resulting JSON file contains the full provenance graph, which you can transform to match your target visualization library's schema. Because the export routine validates the MLMD file presence and pipeline existence, the output is guaranteed to be well-formed JSON suitable for immediate downstream processing.

Consuming Lineage via REST API

D3-Compatible Endpoints

The CMF server exposes specialized REST endpoints that return lineage structures already arranged for D3-tree or D3-force layouts. These endpoints eliminate the need for manual JSON transformation, providing ready-to-render graph data.

According to server/app/main.py, the following endpoints are available:

• GET /execution-lineage/tangled-tree/{uuid}/{pipeline_name} – Returns a D3-compatible node/link graph for a specific execution (lines 20–35).
• GET /artifact-lineage/tangled-tree/{pipeline_name} – Returns a nested list of artifacts with parent-child relationships for the Artifact-Tree view (lines 36–52).
• GET /artifact-execution-lineage/tangled-tree/{pipeline_name} – Returns the combined artifact-execution graph for the Artifact-Execution Tree (lines 21–28).

Query Execution Helpers

Behind these endpoints, the query_execution_lineage_d3tree.py module constructs the execution lineage by:

1. Looking up execution IDs by matching the first 4 characters of the UUID.
2. Walking the parent-execution graph using a unique queue to avoid cycles.
3. Performing a topological sort (lines 75–108) to generate the hierarchical D3-tree format.

Similarly, query_artifact_lineage_d3tree.py handles artifact-specific lineage construction.

Fetching Lineage with curl

Retrieve execution lineage for external visualization using a simple HTTP request:

# Get execution-lineage for execution uuid prefix "Prep" in pipeline "my-pipeline"

curl http://localhost:8000/execution-lineage/tangled-tree/Prep/my-pipeline \
     -o execution_lineage.json

The endpoint returns JSON in the structure { nodes: [{id, name, execution_uuid}], links: [{source, target}] }, which conforms to D3's expected input format.

Python Client Implementation

For programmatic access, use Python's requests library to fetch and save lineage data:

import requests
import json
from pathlib import Path

API_ROOT = "http://localhost:8000"

def fetch_execution_lineage(pipeline, uuid_prefix):
    url = f"{API_ROOT}/execution-lineage/tangled-tree/{uuid_prefix}/{pipeline}"
    resp = requests.get(url)
    resp.raise_for_status()
    return resp.json()          # already in D3-tree format

if __name__ == "__main__":
    pipeline = "my-pipeline"
    uuid = "Prep"               # first 4 chars of the execution UUID

    lineage = fetch_execution_lineage(pipeline, uuid)

    # Save to a file that a D3 front-end can read

    (Path("static") / "execution_lineage.json").write_text(
        json.dumps(lineage, indent=2)
    )
    print("Lineage JSON written – serve `static/` with your D3 page.")

Integrating with External Visualization Tools

Neo4j Graph Database Import

The JSON output from either the CLI or REST API can be imported into Neo4j for complex graph queries and analytics. The node-link structure maps naturally to Cypher's graph model:

// Assuming the JSON payload is saved as execution_lineage.json and loaded via APOC
CALL apoc.load.json("file:///execution_lineage.json") YIELD value AS graph
UNWIND graph.nodes AS n
MERGE (e:Execution {id: n.id})
UNWIND graph.links AS l
MATCH (src:Execution {id: l.source}), (tgt:Execution {id: l.target})
MERGE (src)-[:PRECEDES]->(tgt);

This creates a graph of execution nodes connected by precedence relationships, enabling path-finding and dependency analysis queries.

Custom D3.js Visualizations

The REST endpoints specifically target D3.js visualization libraries. The JSON structure returned from /execution-lineage/tangled-tree/ can be fed directly into D3's tree or force-directed layout components without modification. The docs/ui/lineage.md documentation describes the visualization concepts that you can replicate in your own D3 implementations.

Key Implementation Files

File	Role
cmflib/commands/metadata/export.py	Implements the cmf metadata export CLI command that dumps pipeline metadata to JSON.
server/app/main.py	FastAPI router exposing lineage endpoints (/execution-lineage/*, /artifact-lineage/*).
server/app/query_execution_lineage_d3tree.py	Builds execution-lineage graphs, performs topological sorting, and formats output for D3.
server/app/query_artifact_lineage_d3tree.py	Constructs artifact-lineage trees for the Artifact-Tree view.
docs/ui/lineage.md	Documents the four visualization modes (Artifact Tree, Execution Tree, Artifact-Execution Tree, Force-directed).
cmflib/cmf.py	Contains the high-level metadata_export wrapper function (lines 1835–1853) used by the CLI.

Summary

• Use the CLI (cmf metadata export) to generate self-contained JSON files of complete pipeline metadata for offline analysis.
• Use the REST API to obtain D3-compatible graph structures directly, eliminating the need for manual data transformation.
• Query by UUID prefix (first 4 characters) when fetching specific execution lineages via the REST endpoints.
• Import into external tools like Neo4j using the standard node-link JSON format, or render directly with D3.js using the pre-formatted API responses.
• Reference docs/ui/lineage.md to understand the four visualization modes available for external recreation.

Frequently Asked Questions

What format does CMF use for lineage exports?

CMF exports lineage data as JSON documents. The CLI export produces a comprehensive metadata dump including artifacts, executions, and contexts, while the REST API endpoints return specialized structures formatted as D3-compatible node-link graphs with the schema { nodes: [...], links: [...] }.

How do I find the correct execution UUID prefix for REST queries?

The GET /execution-lineage/tangled-tree/{uuid}/{pipeline_name} endpoint requires only the first 4 characters of the execution UUID. The implementation in query_execution_lineage_d3tree.py filters the execution dataframe using df['Execution_uuid'].str[:4] == uuid, allowing you to query without knowing the full UUID.

Can I export lineage data without running the CMF server?

Yes. The CLI command cmf metadata export operates directly against the MLMD store file without requiring the server to be running. This is ideal for CI/CD pipelines or offline analysis where you only have access to the filesystem containing the metadata database.

Which visualization tools are compatible with CMF lineage exports?

CMF lineage exports work with any tool that accepts JSON graph data, including Neo4j (via Cypher and APOC), D3.js (using the native node-link format), GraphViz (after minor transformation), and Python visualization libraries like NetworkX. The REST API specifically targets D3 tree and force-directed layouts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}