# OpenMAIC Persistence Stack: Architecture, Configuration, and Deployment Options

> Explore the OpenMAIC persistence stack architecture for flexible data storage. Learn configuration and deployment options including browser, server, or hybrid modes.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: architecture
- Published: 2026-09-01

---

**OpenMAIC implements a dual-layer persistence stack that enables applications to store workspace data and assets entirely in the browser via IndexedDB, entirely on the server via PostgreSQL and S3, or in a synchronized hybrid mode controlled by environment variables.**

The OpenMAIC persistence stack is the data foundation of the THU-MAIC/OpenMAIC repository, managing how documents, runtime state, and binary assets are stored across client and server environments. This architecture supports fully offline workflows, centralized server deployments, or mixed configurations where the UI caches data locally while synchronizing with a remote PostgreSQL backend.

## Dual-Layer Architecture Overview

The persistence system is organized into two primary layers that operate independently or in concert. At runtime, the application evaluates environment variables to determine whether to route storage operations to the client-side IndexedDB implementation or to the server-side API routes backed by PostgreSQL.

### Client-Side Storage with IndexedDB

The browser layer persists **stage data** (documents, chat logs, UI state), runtime snapshots, and asset references using **IndexedDB** wrapped by the **Dexie** library. The implementation resides in `lib/utils/stage-storage` and is validated by test suites such as [`tests/store/stage-document-persistence.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/store/stage-document-persistence.test.ts).

Key characteristics of the client layer include:

- **Asynchronous, offline-capable storage** that functions without network connectivity.
- A **memory-only fallback** (`MemoryStorage`) activated when IndexedDB is unavailable.
- A **plain JSON file fallback** ([`plain-json.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/plain-json.ts)) for static-site builds or restricted environments.

### Server-Side PostgreSQL and S3 Storage

The server layer provides authoritative persistence through PostgreSQL tables and S3-compatible object storage. The `@openmaic/storage` packages implement concrete stores (`PgRuntimeStore`, `PgDocumentStore`, `PgAssetStore`) located in `runtime/pg`, `document/pg`, and `asset/pg` directories.

Critical server-side behaviors include:

- **Singleton provider pattern**: [`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts) caches the connection pool using `globalThis[Symbol.for('openmaic.persistence.provider')]` to ensure pool reuse across requests.
- **Automatic schema migration**: Functions like `ensureSchema`, `ensureDocumentSchema`, and `ensureAssetSchema` create tables on demand before the provider accepts requests.
- **Binary asset storage**: The `lazyAssetByteStore` function streams raw asset blobs to the S3 bucket specified in `ASSET_S3_BUCKET`.

### Hybrid Orchestration Mode

When `NEXT_PUBLIC_PERSISTENCE=1` is set on the client, the UI forwards storage calls to `/api/persistence/*` endpoints, which delegate to the server provider. When disabled, the application defaults to the IndexedDB implementation in `lib/utils/stage-storage`, preserving a fully offline experience. In hybrid mode, the client maintains a short-lived IndexedDB cache for responsiveness while treating PostgreSQL as the source of truth.

## Core Components and Source Files

The persistence stack is modularized into specific files responsible for distinct storage concerns:

### Server Provider Singleton

The file [`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts) exports `getServerPersistenceProvider`, which constructs a `ServerPersistenceProvider` containing a PostgreSQL `Pool` and instances of `PgRuntimeStore`, `PgDocumentStore`, and `PgAssetStore`. This provider is cached globally to prevent connection pool exhaustion and ensures all required schemas exist via `ensureSchema` calls before returning the instance.

### Document and Asset Storage

- **[`lib/persistence/owner-bound-document-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/owner-bound-document-store.ts)**: Implements an owner-scoped document store that routes to `PgDocumentStore` when server persistence is active, otherwise delegating to the Dexie client implementation.
- **[`lib/persistence/asset-byte-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/asset-byte-store.ts)**: Lazily initializes a byte store that writes to the configured S3 bucket using `process.env.ASSET_S3_BUCKET`.
- **[`lib/persistence/owner-materials.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/owner-materials.ts)**: Manages per-owner material references with referential integrity guarantees.
- **[`lib/persistence/stage-meta.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/stage-meta.ts)**: Defines the schema for stage metadata (owner identifiers, timestamps) and validates stage IDs.

### Client-Side Stage Storage

The `lib/utils/stage-storage` directory contains the Dexie-based implementation used when operating in offline mode. This module handles stage data serialization, asset reference tracking, and UI state snapshots without requiring server connectivity.

## Configuration Environment Variables

The following variables control persistence behavior across environments:

| Variable | Scope | Purpose | Typical Values |
|----------|-------|---------|----------------|
| `NEXT_PUBLIC_PERSISTENCE` | Client | Enables server-side persistence API routes (`/api/persistence/*`). | `1` = enabled, omitted = offline-only |
| `PERSISTENCE_CONNECTION_STRING` | Server | PostgreSQL connection URL. | `postgres://user:pass@host/db` |
| `ASSET_S3_BUCKET` | Server | Target S3 bucket for binary assets consumed by `lazyAssetByteStore`. | `<bucket-name>` |
| `PERSISTENCE_DEV_TOKEN` | Server (dev) | Authentication token for persistence API access in development. | Random string in `.env` |
| `PERSISTENCE_POOL_FACTORY` | Server | Optional custom factory function for creating `pg.Pool` instances. | `(cs) => new Pool({ connectionString: cs })` |

## Working with the Persistence Stack

The following examples demonstrate how to interact with the persistence layer in server and client contexts.

### Server-Side Provider Usage

```typescript
// lib/persistence/server-provider.ts
import { getServerPersistenceProvider } from '@/lib/persistence/server-provider';

const connStr = process.env.PERSISTENCE_CONNECTION_STRING!;
const provider = await getServerPersistenceProvider(connStr);

// Persist runtime state
await provider.runtimeStore.saveSnapshot({
  sessionId: 'abc123',
  payload: { /* runtime payload */ },
});

```

### Client-Side Local Storage

```typescript
// lib/utils/stage-storage
import { stageStorage } from '@/lib/utils/stage-storage';

await stageStorage.saveStageData('stage/xyz', {
  document: { /* document data */ },
  assets: [/* asset references */],
});

```

### Runtime Mode Detection

```typescript
// Environment-based persistence selection
if (process.env.NEXT_PUBLIC_PERSISTENCE === '1') {
  // UI calls /api/persistence/* routes
  // Server provider handles PostgreSQL writes
} else {
  // All data remains in browser IndexedDB
}

```

## Summary

- **OpenMAIC persistence stack** supports three operational modes: offline-only (IndexedDB), server-only (PostgreSQL/S3), and hybrid (client cache with server sync).
- The **server provider** ([`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts)) is implemented as a singleton using `Symbol.for('openmaic.persistence.provider')` to manage PostgreSQL connection pooling.
- **Client storage** relies on Dexie-backed IndexedDB in `lib/utils/stage-storage`, with automatic fallback to `MemoryStorage` when browser storage is unavailable.
- **Configuration** is controlled via `NEXT_PUBLIC_PERSISTENCE`, `PERSISTENCE_CONNECTION_STRING`, and `ASSET_S3_BUCKET` environment variables.
- All server-side stores perform **automatic schema creation** via `ensureSchema` functions before accepting data operations.

## Frequently Asked Questions

### What storage backends does the OpenMAIC persistence stack support?

The stack supports **IndexedDB** (via Dexie) for client-side storage, **PostgreSQL** for structured runtime and document data, and **S3-compatible object storage** for binary assets. A memory-only fallback and plain JSON file fallback are also available for restricted environments.

### How does OpenMAIC handle offline mode when server persistence is disabled?

When `NEXT_PUBLIC_PERSISTENCE` is omitted or disabled, the application routes all storage operations to `lib/utils/stage-storage`, which uses IndexedDB to persist stage data, documents, and UI state locally. This enables complete offline operation without server connectivity.

### Where is the server-side persistence provider instantiated and cached?

The provider is created in [`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts) by the `getServerPersistenceProvider` function. It is cached as a singleton on `globalThis` using the symbol `Symbol.for('openmaic.persistence.provider')`, ensuring that the same PostgreSQL connection pool is reused across all API requests.

### What environment variables are required to enable server-side persistence?

To activate server persistence, set `NEXT_PUBLIC_PERSISTENCE=1` in the client environment and `PERSISTENCE_CONNECTION_STRING` (PostgreSQL URL) in the server environment. For asset storage, `ASSET_S3_BUCKET` must specify the target S3 bucket name. Optional variables include `PERSISTENCE_DEV_TOKEN` for development authentication and `PERSISTENCE_POOL_FACTORY` for custom connection pooling logic.