# How to Export CMF Lineage Data for External Visualization Tools

> Export CMF lineage data using the CLI or REST API. Generate JSON files or query D3 compatible graphs for Neo4j, GraphViz, and custom visualization tools.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: how-to-guide
- Published: 2026-03-03

---

**You can export CMF lineage data using the `cmf metadata export` CLI command to generate JSON files, or query the REST API endpoints that return D3-compatible graph structures ready for visualization tools like Neo4j, GraphViz, or custom D3 applications.**

The Common Metadata Framework (CMF) captures complete **artifact** and **execution** provenance in an ML Metadata (MLMD) store. While CMF provides a built-in UI for lineage visualization, data science teams often need to export this lineage data to external visualization tools for custom dashboards or graph analysis. This guide covers the two primary export pathways—CLI serialization and REST API consumption—using the actual implementation in the `hewlettpackard/cmf` repository.

## Understanding CMF Lineage Export Architecture

### The MLMD Store Foundation

CMF persists all pipeline metadata—including artifacts, executions, and contexts—inside an **MLMD store**. This provenance data powers the four visualization modes documented in [`docs/ui/lineage.md`](https://github.com/hewlettpackard/cmf/blob/main/docs/ui/lineage.md): Artifact Tree, Execution Tree, Artifact-Execution Tree, and Force-directed layouts. Both export methods ultimately query this same underlying store, but package the results differently for external consumption.

### Export Pathways Overview

The repository provides two authoritative methods for accessing lineage data externally:

1. **CLI Export**: Serialize the entire pipeline metadata to a self-contained JSON file using `cmf metadata export` (implemented in [`cmflib/commands/metadata/export.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/metadata/export.py)).
2. **REST API**: Query purpose-built endpoints that return pre-formatted D3-compatible graphs ( routed in [`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py)).

Both approaches leverage the same query helpers—[`query_execution_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/query_execution_lineage_d3tree.py) and [`query_artifact_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/query_artifact_lineage_d3tree.py)—to construct the lineage structures.

## Exporting Lineage via the CMF CLI

### The CmdMetadataExport Implementation

The command-line interface provides the most straightforward method for bulk exporting lineage data. The `CmdMetadataExport` class in [`cmflib/commands/metadata/export.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/metadata/export.py) handles validation, MLMD store connection, and JSON serialization.

When you invoke the export command, the `run()` method (lines 51–69) instantiates a `cmfquery.CmfQuery` object, then calls `query.dumptojson(pipeline_name, None)` (lines 21–23) to extract the complete metadata graph. The output includes artifacts, executions, contexts, and their relationships in a machine-readable format.

### Step-by-Step CLI Export Workflow

Execute the following to dump a pipeline's metadata:

```bash

# Export the pipeline "my-pipeline" to my-pipeline.json

cmf metadata export -p my-pipeline -j my-pipeline.json

```

The resulting JSON file contains the full provenance graph, which you can transform to match your target visualization library's schema. Because the export routine validates the MLMD file presence and pipeline existence, the output is guaranteed to be well-formed JSON suitable for immediate downstream processing.

## Consuming Lineage via REST API

### D3-Compatible Endpoints

The CMF server exposes specialized REST endpoints that return lineage structures already arranged for D3-tree or D3-force layouts. These endpoints eliminate the need for manual JSON transformation, providing ready-to-render graph data.

According to [`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py), the following endpoints are available:

- `GET /execution-lineage/tangled-tree/{uuid}/{pipeline_name}` – Returns a D3-compatible node/link graph for a specific execution (lines 20–35).
- `GET /artifact-lineage/tangled-tree/{pipeline_name}` – Returns a nested list of artifacts with parent-child relationships for the Artifact-Tree view (lines 36–52).
- `GET /artifact-execution-lineage/tangled-tree/{pipeline_name}` – Returns the combined artifact-execution graph for the Artifact-Execution Tree (lines 21–28).

### Query Execution Helpers

Behind these endpoints, the [`query_execution_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/query_execution_lineage_d3tree.py) module constructs the execution lineage by:

1. Looking up execution IDs by matching the first 4 characters of the UUID.
2. Walking the parent-execution graph using a unique queue to avoid cycles.
3. Performing a topological sort (lines 75–108) to generate the hierarchical D3-tree format.

Similarly, [`query_artifact_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/query_artifact_lineage_d3tree.py) handles artifact-specific lineage construction.

### Fetching Lineage with curl

Retrieve execution lineage for external visualization using a simple HTTP request:

```bash

# Get execution-lineage for execution uuid prefix "Prep" in pipeline "my-pipeline"

curl http://localhost:8000/execution-lineage/tangled-tree/Prep/my-pipeline \
     -o execution_lineage.json

```

The endpoint returns JSON in the structure `{ nodes: [{id, name, execution_uuid}], links: [{source, target}] }`, which conforms to D3's expected input format.

### Python Client Implementation

For programmatic access, use Python's `requests` library to fetch and save lineage data:

```python
import requests
import json
from pathlib import Path

API_ROOT = "http://localhost:8000"

def fetch_execution_lineage(pipeline, uuid_prefix):
    url = f"{API_ROOT}/execution-lineage/tangled-tree/{uuid_prefix}/{pipeline}"
    resp = requests.get(url)
    resp.raise_for_status()
    return resp.json()          # already in D3-tree format

if __name__ == "__main__":
    pipeline = "my-pipeline"
    uuid = "Prep"               # first 4 chars of the execution UUID

    lineage = fetch_execution_lineage(pipeline, uuid)

    # Save to a file that a D3 front-end can read

    (Path("static") / "execution_lineage.json").write_text(
        json.dumps(lineage, indent=2)
    )
    print("Lineage JSON written – serve `static/` with your D3 page.")

```

## Integrating with External Visualization Tools

### Neo4j Graph Database Import

The JSON output from either the CLI or REST API can be imported into **Neo4j** for complex graph queries and analytics. The node-link structure maps naturally to Cypher's graph model:

```cypher
// Assuming the JSON payload is saved as execution_lineage.json and loaded via APOC
CALL apoc.load.json("file:///execution_lineage.json") YIELD value AS graph
UNWIND graph.nodes AS n
MERGE (e:Execution {id: n.id})
UNWIND graph.links AS l
MATCH (src:Execution {id: l.source}), (tgt:Execution {id: l.target})
MERGE (src)-[:PRECEDES]->(tgt);

```

This creates a graph of execution nodes connected by precedence relationships, enabling path-finding and dependency analysis queries.

### Custom D3.js Visualizations

The REST endpoints specifically target **D3.js** visualization libraries. The JSON structure returned from `/execution-lineage/tangled-tree/` can be fed directly into D3's tree or force-directed layout components without modification. The [`docs/ui/lineage.md`](https://github.com/hewlettpackard/cmf/blob/main/docs/ui/lineage.md) documentation describes the visualization concepts that you can replicate in your own D3 implementations.

## Key Implementation Files

| File | Role |
|------|------|
| [`cmflib/commands/metadata/export.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/commands/metadata/export.py) | Implements the `cmf metadata export` CLI command that dumps pipeline metadata to JSON. |
| [`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py) | FastAPI router exposing lineage endpoints (`/execution-lineage/*`, `/artifact-lineage/*`). |
| [`server/app/query_execution_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/query_execution_lineage_d3tree.py) | Builds execution-lineage graphs, performs topological sorting, and formats output for D3. |
| [`server/app/query_artifact_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/query_artifact_lineage_d3tree.py) | Constructs artifact-lineage trees for the Artifact-Tree view. |
| [`docs/ui/lineage.md`](https://github.com/hewlettpackard/cmf/blob/main/docs/ui/lineage.md) | Documents the four visualization modes (Artifact Tree, Execution Tree, Artifact-Execution Tree, Force-directed). |
| [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) | Contains the high-level `metadata_export` wrapper function (lines 1835–1853) used by the CLI. |

## Summary

- **Use the CLI** (`cmf metadata export`) to generate self-contained JSON files of complete pipeline metadata for offline analysis.
- **Use the REST API** to obtain D3-compatible graph structures directly, eliminating the need for manual data transformation.
- **Query by UUID prefix** (first 4 characters) when fetching specific execution lineages via the REST endpoints.
- **Import into external tools** like Neo4j using the standard node-link JSON format, or render directly with D3.js using the pre-formatted API responses.
- **Reference [`docs/ui/lineage.md`](https://github.com/hewlettpackard/cmf/blob/main/docs/ui/lineage.md)** to understand the four visualization modes available for external recreation.

## Frequently Asked Questions

### What format does CMF use for lineage exports?

CMF exports lineage data as **JSON documents**. The CLI export produces a comprehensive metadata dump including artifacts, executions, and contexts, while the REST API endpoints return specialized structures formatted as D3-compatible node-link graphs with the schema `{ nodes: [...], links: [...] }`.

### How do I find the correct execution UUID prefix for REST queries?

The `GET /execution-lineage/tangled-tree/{uuid}/{pipeline_name}` endpoint requires only the **first 4 characters** of the execution UUID. The implementation in [`query_execution_lineage_d3tree.py`](https://github.com/hewlettpackard/cmf/blob/main/query_execution_lineage_d3tree.py) filters the execution dataframe using `df['Execution_uuid'].str[:4] == uuid`, allowing you to query without knowing the full UUID.

### Can I export lineage data without running the CMF server?

**Yes.** The CLI command `cmf metadata export` operates directly against the MLMD store file without requiring the server to be running. This is ideal for CI/CD pipelines or offline analysis where you only have access to the filesystem containing the metadata database.

### Which visualization tools are compatible with CMF lineage exports?

CMF lineage exports work with any tool that accepts **JSON graph data**, including **Neo4j** (via Cypher and APOC), **D3.js** (using the native node-link format), **GraphViz** (after minor transformation), and **Python visualization libraries** like NetworkX. The REST API specifically targets D3 tree and force-directed layouts.