# How to Migrate from SQLite to PostgreSQL for Production CMF Deployments

> Migrate CMF deployments from SQLite to PostgreSQL for production. Configure five POSTGRES environment variables to switch to a scalable PostgreSQL backend.

- Repository: [Hewlett Packard Enterprise/cmf](https://github.com/hewlettpackard/cmf)
- Tags: migration-guide
- Published: 2026-03-03

---

**Set `is_server=True` when initializing the `Cmf` or `CmfQuery` class and configure the five `POSTGRES_*` environment variables to switch from local SQLite files to a scalable PostgreSQL backend.**

The Hewlett Packard Enterprise CMF (Common Metadata Framework) library relies on Google's ML Metadata (MLMD) to track machine learning pipeline lineage. While the default `SqlliteStore` suits individual development, production teams must migrate to PostgreSQL to support concurrent users, centralized access, and durable storage. This guide explains the architecture behind CMF's storage abstraction and provides exact steps to migrate from SQLite to PostgreSQL for production CMF deployments.

## Understanding CMF Storage Architecture

CMF abstracts the underlying metadata store through two concrete implementations selected at runtime. The storage backend is determined entirely by the **`is_server`** initialization flag, which triggers different instantiation logic inside the core library.

### Client-Side SQLite Storage

When `is_server=False` (the default), the `Cmf` class constructor in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) (lines 55‑62) instantiates a `SqlliteStore` pointing to a local file. This mode writes `mlmd` SQLite databases to the client filesystem, suitable for single-user experimentation but inadequate for multi-user production environments.

### Server-Side PostgreSQL Storage

When `is_server=True`, the constructor invokes `get_postgres_config()` from [`cmflib/utils/helper_functions.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/helper_functions.py) (lines 75‑88) to read database credentials from environment variables. It then instantiates the `PostgresStore` class defined in [`cmflib/store/postgres.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/store/postgres.py) (lines 8‑15). This store connects to a remote PostgreSQL instance, providing the concurrency and durability required for production CMF server deployments.

## Production Deployment Configuration

The CMF repository includes a Docker‑Compose orchestration that automates PostgreSQL provisioning and environment injection.

### Docker-Compose PostgreSQL Service

The [`docker-compose-server.yml`](https://github.com/hewlettpackard/cmf/blob/main/docker-compose-server.yml) file (lines 18‑30) defines a PostgreSQL service with persistent volume storage and a health check. The CMF server container declares an explicit `depends_on` constraint to ensure PostgreSQL is healthy before startup. This configuration eliminates manual database installation and networking setup.

### Environment Variable Configuration

Create a `.env` file adjacent to [`docker-compose-server.yml`](https://github.com/hewlettpackard/cmf/blob/main/docker-compose-server.yml) with the following variables as documented in [`docs/setup/index.md`](https://github.com/hewlettpackard/cmf/blob/main/docs/setup/index.md) (lines 8‑15):

```bash
POSTGRES_HOST=postgres
POSTGRES_PORT=5432
POSTGRES_USER=myuser
POSTGRES_PASSWORD=mypassword
POSTGRES_DB=mlmd
REACT_APP_CMF_API_URL=http://<server-ip>:80

```

The CMF server automatically ingests these values through `get_postgres_config()` on initialization. No code changes are required—only environment configuration.

## Step-by-Step Migration Process

Follow these steps to migrate existing metadata from SQLite to PostgreSQL without losing lineage history.

1. **Provision PostgreSQL**

   Use the provided compose file to start PostgreSQL:  
   `docker compose -f docker-compose-server.yml up postgres -d`  
   Alternatively, point to an existing external PostgreSQL instance by setting the `POSTGRES_*` variables in your environment.

2. **Configure the CMF Server**

   Populate the `.env` file with connection credentials. The server will connect to this database instead of looking for local SQLite files.

3. **Start the Full Stack**

   Run `docker compose -f docker-compose-server.yml up` to launch PostgreSQL, the CMF server, UI, TensorBoard, and Nginx. The server waits for PostgreSQL health checks to pass before accepting connections.

4. **Migrate Existing SQLite Data**

   Because both stores share the MLMD schema, you can export objects from SQLite and recreate them in PostgreSQL. Iterate over contexts, artifacts, and executions using the client‑side API, then push them to the server-side store.

5. **Update Client Code**

   Scripts that log metadata can continue using `Cmf(filepath="mlmd")` for local buffering. For querying the central store, instantiate `CmfQuery(is_server=True)` as shown in [`server/app/main.py`](https://github.com/hewlettpackard/cmf/blob/main/server/app/main.py) (line 60).

6. **Verify Connectivity**

   Access the web UI at `http://<server-ip>:80` and execute a `CmfQuery` retrieval to confirm that executions and artifacts persist in PostgreSQL.

## Code Examples

### Development Client Writing to SQLite

```python
from cmflib.cmf import Cmf

# Creates mlmd SQLite file in current directory

logger = Cmf(filepath="mlmd", pipeline_name="my_pipeline")
ctx = logger.create_context("prepare")

# ... log artifacts, metrics, executions

logger.finalize()

```

### Production Query Against PostgreSQL

```python
from cmflib.cmfquery import CmfQuery

# Requires POSTGRES_* environment variables

query = CmfQuery(is_server=True)

# Fetch executions from central PostgreSQL store

executions = query.get_executions(pipeline_name="my_pipeline")
for exe in executions:
    print(exe.id, exe.name)

```

### One-Off Metadata Migration Script

```python

# Step A: Read from SQLite source

from cmflib.cmf import Cmf
src = Cmf(filepath="mlmd", pipeline_name="temp", is_server=False)

contexts = src.store.get_contexts()
artifacts = src.store.get_artifacts()
executions = src.store.get_executions()

# Step B: Write to PostgreSQL destination

from cmflib.cmfquery import CmfQuery
dst = CmfQuery(is_server=True)

# Recreate objects in PostgreSQL

for ctx in contexts:
    dst.store.put_context(ctx)
for art in artifacts:
    dst.store.put_artifact(art)
for exe in executions:
    dst.store.put_execution(exe)

```

## Summary

- **Storage selection** is controlled by the `is_server` boolean flag in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py).
- **SQLite** (`is_server=False`) creates local `mlmd` files suitable only for development.
- **PostgreSQL** (`is_server=True`) requires `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB` environment variables read by `get_postgres_config()`.
- **Migration** involves iterating over objects in the SQLite store via `Cmf` and writing them to the PostgreSQL store via `CmfQuery`.
- **Production deployment** uses [`docker-compose-server.yml`](https://github.com/hewlettpackard/cmf/blob/main/docker-compose-server.yml) to orchestrate PostgreSQL, the CMF server, and the UI.

## Frequently Asked Questions

### What environment variables are required to connect CMF to PostgreSQL?

CMF requires five variables: `POSTGRES_HOST`, `POSTGRES_PORT`, `POSTGRES_USER`, `POSTGRES_PASSWORD`, and `POSTGRES_DB`. The function `get_postgres_config()` in [`cmflib/utils/helper_functions.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/utils/helper_functions.py) reads these at runtime to construct the `PostgresStore` connection string.

### Can I migrate existing SQLite metadata to PostgreSQL without data loss?

Yes. Since both `SqlliteStore` and `PostgresStore` implement the same MLMD schema, you can export contexts, artifacts, and executions from the SQLite-backed `Cmf` instance and recreate them using the `CmfQuery` instance connected to PostgreSQL. The raw analysis provides a sample migration script that iterates over `src.store.get_contexts()` and calls `dst.store.put_context()`.

### How does the CMF server handle database connections differently than the client?

The CMF client defaults to `is_server=False`, which instantiates a local `SqlliteStore` in [`cmflib/cmf.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/cmf.py) (lines 55‑62). The production server sets `is_server=True`, triggering the constructor to load PostgreSQL credentials and instantiate `PostgresStore` from [`cmflib/store/postgres.py`](https://github.com/hewlettpackard/cmf/blob/main/cmflib/store/postgres.py), enabling centralized, concurrent access.

### Is Docker required to run CMF with PostgreSQL?

No. While the repository provides [`docker-compose-server.yml`](https://github.com/hewlettpackard/cmf/blob/main/docker-compose-server.yml) to simplify deployment, you can run the CMF server against any accessible PostgreSQL instance by setting the required `POSTGRES_*` environment variables in your shell or configuration management system before starting the Python process.