OpenMAIC Persistence Stack: Architecture, Configuration, and Deployment Options

OpenMAIC implements a dual-layer persistence stack that enables applications to store workspace data and assets entirely in the browser via IndexedDB, entirely on the server via PostgreSQL and S3, or in a synchronized hybrid mode controlled by environment variables.

The OpenMAIC persistence stack is the data foundation of the THU-MAIC/OpenMAIC repository, managing how documents, runtime state, and binary assets are stored across client and server environments. This architecture supports fully offline workflows, centralized server deployments, or mixed configurations where the UI caches data locally while synchronizing with a remote PostgreSQL backend.

Dual-Layer Architecture Overview

The persistence system is organized into two primary layers that operate independently or in concert. At runtime, the application evaluates environment variables to determine whether to route storage operations to the client-side IndexedDB implementation or to the server-side API routes backed by PostgreSQL.

Client-Side Storage with IndexedDB

The browser layer persists stage data (documents, chat logs, UI state), runtime snapshots, and asset references using IndexedDB wrapped by the Dexie library. The implementation resides in lib/utils/stage-storage and is validated by test suites such as tests/store/stage-document-persistence.test.ts.

Key characteristics of the client layer include:

• Asynchronous, offline-capable storage that functions without network connectivity.
• A memory-only fallback (MemoryStorage) activated when IndexedDB is unavailable.
• A plain JSON file fallback (plain-json.ts) for static-site builds or restricted environments.

Server-Side PostgreSQL and S3 Storage

The server layer provides authoritative persistence through PostgreSQL tables and S3-compatible object storage. The @openmaic/storage packages implement concrete stores (PgRuntimeStore, PgDocumentStore, PgAssetStore) located in runtime/pg, document/pg, and asset/pg directories.

Critical server-side behaviors include:

• Singleton provider pattern: lib/persistence/server-provider.ts caches the connection pool using globalThis[Symbol.for('openmaic.persistence.provider')] to ensure pool reuse across requests.
• Automatic schema migration: Functions like ensureSchema, ensureDocumentSchema, and ensureAssetSchema create tables on demand before the provider accepts requests.
• Binary asset storage: The lazyAssetByteStore function streams raw asset blobs to the S3 bucket specified in ASSET_S3_BUCKET.

Hybrid Orchestration Mode

When NEXT_PUBLIC_PERSISTENCE=1 is set on the client, the UI forwards storage calls to /api/persistence/* endpoints, which delegate to the server provider. When disabled, the application defaults to the IndexedDB implementation in lib/utils/stage-storage, preserving a fully offline experience. In hybrid mode, the client maintains a short-lived IndexedDB cache for responsiveness while treating PostgreSQL as the source of truth.

Core Components and Source Files

The persistence stack is modularized into specific files responsible for distinct storage concerns:

Server Provider Singleton

The file lib/persistence/server-provider.ts exports getServerPersistenceProvider, which constructs a ServerPersistenceProvider containing a PostgreSQL Pool and instances of PgRuntimeStore, PgDocumentStore, and PgAssetStore. This provider is cached globally to prevent connection pool exhaustion and ensures all required schemas exist via ensureSchema calls before returning the instance.

Document and Asset Storage

• lib/persistence/owner-bound-document-store.ts: Implements an owner-scoped document store that routes to PgDocumentStore when server persistence is active, otherwise delegating to the Dexie client implementation.
• lib/persistence/asset-byte-store.ts: Lazily initializes a byte store that writes to the configured S3 bucket using process.env.ASSET_S3_BUCKET.
• lib/persistence/owner-materials.ts: Manages per-owner material references with referential integrity guarantees.
• lib/persistence/stage-meta.ts: Defines the schema for stage metadata (owner identifiers, timestamps) and validates stage IDs.

Client-Side Stage Storage

The lib/utils/stage-storage directory contains the Dexie-based implementation used when operating in offline mode. This module handles stage data serialization, asset reference tracking, and UI state snapshots without requiring server connectivity.

Configuration Environment Variables

The following variables control persistence behavior across environments:

Variable	Scope	Purpose	Typical Values
NEXT_PUBLIC_PERSISTENCE	Client	Enables server-side persistence API routes (/api/persistence/*).	1 = enabled, omitted = offline-only
PERSISTENCE_CONNECTION_STRING	Server	PostgreSQL connection URL.	postgres://user:pass@host/db
ASSET_S3_BUCKET	Server	Target S3 bucket for binary assets consumed by lazyAssetByteStore.	<bucket-name>
PERSISTENCE_DEV_TOKEN	Server (dev)	Authentication token for persistence API access in development.	Random string in .env
PERSISTENCE_POOL_FACTORY	Server	Optional custom factory function for creating pg.Pool instances.	(cs) => new Pool({ connectionString: cs })

Working with the Persistence Stack

The following examples demonstrate how to interact with the persistence layer in server and client contexts.

Server-Side Provider Usage

// lib/persistence/server-provider.ts
import { getServerPersistenceProvider } from '@/lib/persistence/server-provider';

const connStr = process.env.PERSISTENCE_CONNECTION_STRING!;
const provider = await getServerPersistenceProvider(connStr);

// Persist runtime state
await provider.runtimeStore.saveSnapshot({
  sessionId: 'abc123',
  payload: { /* runtime payload */ },
});

Client-Side Local Storage

// lib/utils/stage-storage
import { stageStorage } from '@/lib/utils/stage-storage';

await stageStorage.saveStageData('stage/xyz', {
  document: { /* document data */ },
  assets: [/* asset references */],
});

Runtime Mode Detection

// Environment-based persistence selection
if (process.env.NEXT_PUBLIC_PERSISTENCE === '1') {
  // UI calls /api/persistence/* routes
  // Server provider handles PostgreSQL writes
} else {
  // All data remains in browser IndexedDB
}

Summary

• OpenMAIC persistence stack supports three operational modes: offline-only (IndexedDB), server-only (PostgreSQL/S3), and hybrid (client cache with server sync).
• The server provider (lib/persistence/server-provider.ts) is implemented as a singleton using Symbol.for('openmaic.persistence.provider') to manage PostgreSQL connection pooling.
• Client storage relies on Dexie-backed IndexedDB in lib/utils/stage-storage, with automatic fallback to MemoryStorage when browser storage is unavailable.
• Configuration is controlled via NEXT_PUBLIC_PERSISTENCE, PERSISTENCE_CONNECTION_STRING, and ASSET_S3_BUCKET environment variables.
• All server-side stores perform automatic schema creation via ensureSchema functions before accepting data operations.

Frequently Asked Questions

What storage backends does the OpenMAIC persistence stack support?

The stack supports IndexedDB (via Dexie) for client-side storage, PostgreSQL for structured runtime and document data, and S3-compatible object storage for binary assets. A memory-only fallback and plain JSON file fallback are also available for restricted environments.

How does OpenMAIC handle offline mode when server persistence is disabled?

When NEXT_PUBLIC_PERSISTENCE is omitted or disabled, the application routes all storage operations to lib/utils/stage-storage, which uses IndexedDB to persist stage data, documents, and UI state locally. This enables complete offline operation without server connectivity.

Where is the server-side persistence provider instantiated and cached?

The provider is created in lib/persistence/server-provider.ts by the getServerPersistenceProvider function. It is cached as a singleton on globalThis using the symbol Symbol.for('openmaic.persistence.provider'), ensuring that the same PostgreSQL connection pool is reused across all API requests.

What environment variables are required to enable server-side persistence?

To activate server persistence, set NEXT_PUBLIC_PERSISTENCE=1 in the client environment and PERSISTENCE_CONNECTION_STRING (PostgreSQL URL) in the server environment. For asset storage, ASSET_S3_BUCKET must specify the target S3 bucket name. Optional variables include PERSISTENCE_DEV_TOKEN for development authentication and PERSISTENCE_POOL_FACTORY for custom connection pooling logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}