How to Migrate from SQLite to PostgreSQL for Production CMF Deployments

Set is_server=True when initializing the Cmf or CmfQuery class and configure the five POSTGRES_* environment variables to switch from local SQLite files to a scalable PostgreSQL backend.

The Hewlett Packard Enterprise CMF (Common Metadata Framework) library relies on Google's ML Metadata (MLMD) to track machine learning pipeline lineage. While the default SqlliteStore suits individual development, production teams must migrate to PostgreSQL to support concurrent users, centralized access, and durable storage. This guide explains the architecture behind CMF's storage abstraction and provides exact steps to migrate from SQLite to PostgreSQL for production CMF deployments.

Understanding CMF Storage Architecture

CMF abstracts the underlying metadata store through two concrete implementations selected at runtime. The storage backend is determined entirely by the is_server initialization flag, which triggers different instantiation logic inside the core library.

Client-Side SQLite Storage

When is_server=False (the default), the Cmf class constructor in cmflib/cmf.py (lines 55‑62) instantiates a SqlliteStore pointing to a local file. This mode writes mlmd SQLite databases to the client filesystem, suitable for single-user experimentation but inadequate for multi-user production environments.

Server-Side PostgreSQL Storage

When is_server=True, the constructor invokes get_postgres_config() from cmflib/utils/helper_functions.py (lines 75‑88) to read database credentials from environment variables. It then instantiates the PostgresStore class defined in cmflib/store/postgres.py (lines 8‑15). This store connects to a remote PostgreSQL instance, providing the concurrency and durability required for production CMF server deployments.

Production Deployment Configuration

The CMF repository includes a Docker‑Compose orchestration that automates PostgreSQL provisioning and environment injection.

Docker-Compose PostgreSQL Service

The docker-compose-server.yml file (lines 18‑30) defines a PostgreSQL service with persistent volume storage and a health check. The CMF server container declares an explicit depends_on constraint to ensure PostgreSQL is healthy before startup. This configuration eliminates manual database installation and networking setup.

Environment Variable Configuration

Create a .env file adjacent to docker-compose-server.yml with the following variables as documented in docs/setup/index.md (lines 8‑15):

POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=myuser
POSTGRES_PASSWORD=mypassword
POSTGRES_DB=mlmd
REACT_APP_CMF_API_URL=http://<server-ip>:80

The CMF server automatically ingests these values through get_postgres_config() on initialization. No code changes are required—only environment configuration.

Step-by-Step Migration Process

Follow these steps to migrate existing metadata from SQLite to PostgreSQL without losing lineage history.

1. Provision PostgreSQL

   Use the provided compose file to start PostgreSQL:
   docker compose -f docker-compose-server.yml up postgres -d
   Alternatively, point to an existing external PostgreSQL instance by setting the POSTGRES_* variables in your environment.

2. Configure the CMF Server

   Populate the .env file with connection credentials. The server will connect to this database instead of looking for local SQLite files.

3. Start the Full Stack

   Run docker compose -f docker-compose-server.yml up to launch PostgreSQL, the CMF server, UI, TensorBoard, and Nginx. The server waits for PostgreSQL health checks to pass before accepting connections.

4. Migrate Existing SQLite Data

   Because both stores share the MLMD schema, you can export objects from SQLite and recreate them in PostgreSQL. Iterate over contexts, artifacts, and executions using the client‑side API, then push them to the server-side store.

5. Update Client Code

   Scripts that log metadata can continue using Cmf(filepath="mlmd") for local buffering. For querying the central store, instantiate CmfQuery(is_server=True) as shown in server/app/main.py (line 60).

6. Verify Connectivity

   Access the web UI at http://<server-ip>:80 and execute a CmfQuery retrieval to confirm that executions and artifacts persist in PostgreSQL.

Code Examples

Development Client Writing to SQLite

from cmflib.cmf import Cmf

# Creates mlmd SQLite file in current directory

logger = Cmf(filepath="mlmd", pipeline_name="my_pipeline")
ctx = logger.create_context("prepare")

# ... log artifacts, metrics, executions

logger.finalize()

Production Query Against PostgreSQL

from cmflib.cmfquery import CmfQuery

# Requires POSTGRES_* environment variables

query = CmfQuery(is_server=True)

# Fetch executions from central PostgreSQL store

executions = query.get_executions(pipeline_name="my_pipeline")
for exe in executions:
    print(exe.id, exe.name)

One-Off Metadata Migration Script

# Step A: Read from SQLite source

from cmflib.cmf import Cmf
src = Cmf(filepath="mlmd", pipeline_name="temp", is_server=False)

contexts = src.store.get_contexts()
artifacts = src.store.get_artifacts()
executions = src.store.get_executions()

# Step B: Write to PostgreSQL destination

from cmflib.cmfquery import CmfQuery
dst = CmfQuery(is_server=True)

# Recreate objects in PostgreSQL

for ctx in contexts:
    dst.store.put_context(ctx)
for art in artifacts:
    dst.store.put_artifact(art)
for exe in executions:
    dst.store.put_execution(exe)

Summary

• Storage selection is controlled by the is_server boolean flag in cmflib/cmf.py.
• SQLite (is_server=False) creates local mlmd files suitable only for development.
• PostgreSQL (is_server=True) requires POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB environment variables read by get_postgres_config().
• Migration involves iterating over objects in the SQLite store via Cmf and writing them to the PostgreSQL store via CmfQuery.
• Production deployment uses docker-compose-server.yml to orchestrate PostgreSQL, the CMF server, and the UI.

Frequently Asked Questions

What environment variables are required to connect CMF to PostgreSQL?

CMF requires five variables: POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB. The function get_postgres_config() in cmflib/utils/helper_functions.py reads these at runtime to construct the PostgresStore connection string.

Can I migrate existing SQLite metadata to PostgreSQL without data loss?

Yes. Since both SqlliteStore and PostgresStore implement the same MLMD schema, you can export contexts, artifacts, and executions from the SQLite-backed Cmf instance and recreate them using the CmfQuery instance connected to PostgreSQL. The raw analysis provides a sample migration script that iterates over src.store.get_contexts() and calls dst.store.put_context().

How does the CMF server handle database connections differently than the client?

The CMF client defaults to is_server=False, which instantiates a local SqlliteStore in cmflib/cmf.py (lines 55‑62). The production server sets is_server=True, triggering the constructor to load PostgreSQL credentials and instantiate PostgresStore from cmflib/store/postgres.py, enabling centralized, concurrent access.

Is Docker required to run CMF with PostgreSQL?

No. While the repository provides docker-compose-server.yml to simplify deployment, you can run the CMF server against any accessible PostgreSQL instance by setting the required POSTGRES_* environment variables in your shell or configuration management system before starting the Python process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}