How Neo4j Graph Visualization Tracks Pipeline Lineage and Dependencies in CMF

CMF maintains a live Neo4j graph mirror of all ML pipeline operations, enabling real-time visualization of lineage and dependencies through D3.js while keeping the graph synchronized with the ML Metadata store.

The Continuous Metadata Framework (CMF) by Hewlett Packard Enterprise solves the challenge of tracking complex machine learning pipeline provenance by implementing a dual-backend architecture that combines structured metadata storage with graph database capabilities. When enabled, Neo4j graph visualization tracks pipeline lineage and dependencies by mirroring every logging operation—from pipeline creation to artifact generation—as nodes and relationships in a live graph database. This architecture allows data scientists to explore execution history through interactive visualizations while maintaining the performance benefits of the ML Metadata (MLMD) store for structured queries.

Dual-Backend Architecture Overview

CMF stores every piece of ML pipeline metadata in two complementary back-ends that serve different query patterns:

• ML Metadata (MLMD) SQLite/Postgres: Holds structured tables describing pipelines, stages, executions, artifacts, and metrics. This backend provides fast query-by-ID operations for the Python API and server endpoints.
• Neo4j Graph Database: Contains nodes and relationships that mirror the same entities (Pipeline, Stage, Execution, Dataset, Model, Metrics) and the "produced-by/consumes-by" links. This enables visual, graph-oriented exploration of lineage and dependency DAGs in the UI.

The Neo4j graph acts as a live mirror that reflects every write operation performed through the CMF Python API, ensuring the visualization layer always represents the current state of pipeline dependencies.

Writing Pipeline Metadata to Neo4j

When a user initializes a CMF object with graph=True, the library executes a sequence of operations to persist metadata as a graph according to the hewlettpackard/cmf source code.

Loading Connection Parameters

First, CMF loads Neo4j connection parameters from the configuration file or environment variables. In cmflib/cmf.py, the private method __load_neo4j_params handles this initialization:

# cmflib/cmf.py – load Neo4j params (lines 15-23)

Cmf.__load_neo4j_params()

Initializing the GraphDriver

After loading credentials, CMF instantiates the GraphDriver class from graph_wrapper.py, which manages the Neo4j driver connection:

# cmflib/cmf.py – GraphDriver instantiation (lines 95-99)

self.driver = graph_wrapper.GraphDriver(
    Cmf.__neo4j_uri, Cmf.__neo4j_user, Cmf.__neo4j_password
)

Creating the Pipeline Root Node

Upon pipeline initialization, CMF creates a root Pipeline node that serves as the parent for all subsequent operations:

# cmflib/cmf.py – Pipeline node creation (lines 199-202)

self.driver.create_pipeline_node(
    self.pipeline_name, self.parent_context.id, custom_properties
)

Logging Stages, Executions, and Artifacts

Every subsequent CMF call creates or updates Neo4j nodes and relationships through GraphDriver methods. The wrapper provides specific methods for each entity type:

• create_stage_node
• create_execution_node
• create_dataset_node
• create_model_node
• create_execution_artifacts_link_syntax

For example, when logging an artifact, the driver creates the appropriate execution-artifact link using Cypher syntax generation:

# cmflib/graph_wrapper.py – Execution-artifact link (lines 69-77)

pc_syntax = self._create_execution_artifacts_link_syntax(
    "Execution", "Dataset", self.execution_id, node_id, event
)

These operations execute in real-time during normal CMF logging calls, ensuring the graph database stays synchronized with the MLMD store without requiring batch exports.

Querying Lineage for Visualization

While Neo4j stores the graph structure, the CMF server queries the MLMD store to build lineage responses, guaranteeing consistency with the ground-truth metadata.

Building the Dependency DAG

The FastAPI server does not query Neo4j directly for lineage requests. Instead, it uses CmfQuery to extract parent-child relationships from MLMD, then processes them in server/app/query_artifact_lineage_d3tree.py. This module performs topological sorting to guarantee that parent artifacts appear before children in the response:

# server/app/query_artifact_lineage_d3tree.py (lines 22-27)

data_organized = topological_sort(child_parent_artifact_id, id_name)

This topological sort produces a DAG structure that D3.js can render as a tangled tree or force-directed graph.

Serving Lineage via REST Endpoints

The server exposes lineage data through dedicated endpoints defined in server/app/main.py. The /artifact-lineage/tangled-tree/{pipeline_name} endpoint returns a JSON payload formatted for D3 visualization:

# server/app/main.py – Lineage endpoint (lines 37-53)

@app.get("/artifact-lineage/tangled-tree/{pipeline_name}")

Rendering with D3.js

The frontend fetches the processed lineage data and renders it using D3.js. In ui/src/client.js, the application retrieves the JSON structure and passes it to the visualization layer:

// ui/src/client.js (lines 78-82)
.get(`/artifact-lineage/tangled-tree/${pipeline}`)

This decoupled approach—writing to Neo4j while reading from MLMD for visualization—ensures that the UI displays accurate dependency graphs even if the Neo4j driver operates asynchronously.

Practical Implementation Examples

Initializing CMF with Neo4j Graph Enabled

To enable graph visualization, initialize the CMF logger with the graph=True parameter:

from cmflib.cmf import Cmf

# graph=True activates Neo4j writes

cmf_logger = Cmf(
    filepath="mlmd",               # MLMD store

    pipeline_name="my_pipeline",
    graph=True                     # enable Neo4j graph persistence

)

Logging a Stage and Dataset Artifact

When you create stages and log artifacts, the driver automatically writes corresponding nodes and relationships to Neo4j:

# Create a new stage (adds a Stage node + Pipeline->Stage relationship)

stage_ctx = cmf_logger.create_context(pipeline_stage="preprocess")

# Log a dataset artifact (creates a Dataset node + Execution->Dataset link)

cmf_logger.log_dataset_with_version(
    name="raw_data",
    uri="file:///data/raw.csv",
    version="v1",
    event="output",                # marks the dataset as an output

    custom_properties={"source": "ingest"}
)

Retrieving Lineage via REST API

Query the lineage endpoint to retrieve the D3-compatible JSON structure:

curl -s http://localhost:8000/artifact-lineage/tangled-tree/my_pipeline \
  | jq '.'   # pretty-print the JSON D3 payload

The response contains groups of nodes with parent IDs, ready for immediate visualization.

Frontend Integration

The CMF web UI automatically handles the rendering. The client fetches data and invokes the drawing functions:

// client.js – invoked when the user selects a pipeline
fetch(`/artifact-lineage/tangled-tree/${pipeline}`)
  .then(r => r.json())
  .then(data => drawArtifactTree(data));   // D3 force-directed or tangled tree

Summary

• CMF implements a dual-backend architecture combining MLMD for structured storage and Neo4j for graph visualization.
• The GraphDriver class in cmflib/graph_wrapper.py manages real-time writes to Neo4j through methods like create_pipeline_node and create_execution_artifacts_link_syntax.
• Lineage queries are served from MLMD, not Neo4j, ensuring consistency while the graph database acts as a visualization mirror.
• Topological sorting in query_artifact_lineage_d3tree.py guarantees correct DAG ordering for D3.js rendering.
• The FastAPI server exposes endpoints like /artifact-lineage/tangled-tree/{pipeline_name} that return JSON formatted for immediate visualization.

Frequently Asked Questions

How does CMF ensure the Neo4j graph stays synchronized with the MLMD store?

CMF writes to both backends simultaneously during logging operations. When you call methods like log_dataset_with_version or create_context, the GraphDriver executes Cypher queries to create nodes and relationships in Neo4j at the same time it writes records to MLMD. This tight coupling ensures the graph database reflects the exact state of pipeline executions and artifact dependencies without requiring separate synchronization jobs.

Why does the server query MLMD instead of Neo4j for lineage visualization?

The server queries MLMD because it represents the ground-truth metadata store that is always guaranteed to be consistent with the Python API operations. While Neo4j receives the same writes, the MLMD store provides optimized query patterns for parent-child relationships and ensures that the lineage API returns accurate results even if Neo4j connection issues occur. The Neo4j graph serves primarily as a visualization mirror and exploration interface.

What types of relationships does CMF track in Neo4j?

CMF tracks produced-by and consumes-by relationships between executions and artifacts, along with hierarchical links between pipelines, stages, and executions. Specifically, the create_execution_artifacts_link_syntax method generates Cypher relationships connecting Execution nodes to Dataset, Model, and Metrics nodes, enabling the visualization of complete data provenance chains from raw inputs through model training to final evaluations.

Can I query the Neo4j graph directly for custom analysis?

Yes, since Neo4j contains a complete mirror of the pipeline metadata as nodes and relationships, you can connect directly to the Neo4j database using any Cypher-compatible client to perform custom graph analytics. The node labels follow CMF conventions (Pipeline, Stage, Execution, Dataset, Model, Metrics), allowing you to write Cypher queries to find patterns like circular dependencies, most-used datasets, or critical path analysis across pipeline stages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}