Swappable Stores in OpenMAIC's Persistence Layer: 8 Store Types Explained

OpenMAIC's persistence layer provides eight swappable store abstractions—Document, Runtime, Asset, Asset Byte, Owner-bound Document, Owner Materials, Stage Meta, and Plain-JSON stores—allowing seamless switching between PostgreSQL, S3, and IndexedDB backends through interface-based dependency injection.

OpenMAIC implements a pluggable persistence architecture where storage backends can be exchanged without modifying business logic. The THU-MAIC/OpenMAIC repository achieves this through interface-driven design: consuming code receives store abstractions rather than concrete implementations, enabling deployment flexibility across server-side PostgreSQL, client-side IndexedDB, or custom JSON-compatible backends.

---

Core Server-Side Stores (PostgreSQL-Backed)

The primary production stores reside in lib/persistence/server-provider.ts and wrap PostgreSQL for durable persistence.

Document Store: PgDocumentStore

The Document Store persists long-lived entities including scenes and stages. Defined at lines 1-3 of server-provider.ts, PgDocumentStore implements the document storage interface for relational data.

Runtime Store: PgRuntimeStore

Sessions in progress require ephemeral state tracking. The Runtime Store (PgRuntimeStore, lines 20-21) maintains execution context for active workflows, separate from durable documents to optimize query patterns.

Asset Store: PgAssetStore

Binary metadata lives in PgAssetStore (line 22), which tracks large assets such as images and videos while delegating actual byte storage to specialized handlers.

---

Specialized and S3-Backed Stores

Beyond core PostgreSQL stores, OpenMAIC provides targeted implementations for specific access patterns.

Asset Byte Store: lazyAssetByteStore

Actual asset content streams from S3 through the Asset Byte Store. Located in lib/persistence/asset-byte-store.ts, this store provides byte-level access to objects stored in S3 buckets—decoupling metadata (PostgreSQL) from payload storage (object storage) for cost and performance optimization.

Owner-Bound Document Store: OwnerBoundDocumentStore

Multi-tenancy requires data isolation. The Owner-bound Document Store (lib/persistence/owner-bound-document-store.ts) scopes all document operations to a specific user/owner, enforcing access control at the persistence layer.

Owner Materials Store: OwnerMaterials

User-uploaded content—PDFs, slides, and other attachments—reside in the Owner Materials Store (lib/persistence/owner-materials.ts). This PostgreSQL-backed store manages metadata and references for per-user material libraries.

Stage Meta Store: StageMeta

Workflow metadata (creation timestamps, execution status, configuration snapshots) flows through the Stage Meta Store in lib/persistence/stage-meta.ts, enabling audit trails and state reconstruction.

---

Client-Side and Canonicalization Stores

Browser environments and testing scenarios require alternative implementations.

Client-Side IndexedDB Stores

When NEXT_PUBLIC_PERSISTENCE=0, OpenMAIC loads Dexie-based IndexedDB stores for offline-first capability. Key client implementations include:

• lib/persistence/document-access.ts — IndexedDB Document Store
• lib/persistence/asset-byte-store.ts (client version) — IndexedDB Asset Byte Store

These provide identical interfaces to their server counterparts, enabling code sharing between server and browser contexts.

Plain-JSON Store: omitUndefinedObjectMembers

The lib/persistence/plain-json.ts utility (lines 18-30) implements a canonicalization store that transforms arbitrary objects into JSON-safe representations. This enables:

• Testing with in-memory JSON structures
• Migration scripts with deterministic output
• Custom backend adapters requiring plain objects

import { omitUndefinedObjectMembers } from '@/lib/persistence/plain-json';

const safeObj = omitUndefinedObjectMembers(complexState);
// safeObj can now be passed to any persistence store expecting plain JSON

---

Store Instantiation Patterns

Server-Side Provider Creation

The getServerPersistenceProvider function (lines 70-90 of server-provider.ts) lazily initializes PostgreSQL stores with connection pooling:

import { getServerPersistenceProvider } from '@/lib/persistence/server-provider';

const provider = await getServerPersistenceProvider(process.env.PG_CONNECTION_STRING);
// provider.documentStore, provider.runtimeStore, provider.assetStore

Client-Side Store Creation

Browser environments instantiate Dexie-backed equivalents:

import { createDocumentStore } from '@/lib/persistence/document-access';

const docStore = await createDocumentStore(); // Dexie-backed IndexedDB
await docStore.saveDocument(myDoc);

---

Swapping Store Implementations

The architecture enables swapping through three mechanisms:

1. Environment variables — NEXT_PUBLIC_PERSISTENCE=0 triggers client store loading
2. Provider injection — getServerPersistenceProvider accepts connection strings for different PostgreSQL instances
3. Interface compliance — Custom stores implement the same TypeScript interfaces as PgDocumentStore, PgRuntimeStore, et al.

The plain-JSON canonicalizer serves as the universal adapter: any backend accepting JSON can receive pre-processed data without store-specific transformations.

---

Summary

• OpenMAIC provides eight swappable store types covering documents, runtime state, assets, owner-scoped data, materials, stage metadata, and JSON canonicalization
• Server-side stores (PgDocumentStore, PgRuntimeStore, PgAssetStore) reside in lib/persistence/server-provider.ts
• S3 integration occurs through lazyAssetByteStore in lib/persistence/asset-byte-store.ts
• Multi-tenant isolation uses OwnerBoundDocumentStore and OwnerMaterials for per-user data boundaries
• Client-side offline support comes from Dexie-based IndexedDB stores activated via environment configuration
• Testing and migration leverage omitUndefinedObjectMembers for backend-agnostic JSON handling

---

Frequently Asked Questions

How does OpenMAIC switch between PostgreSQL and IndexedDB stores?

OpenMAIC uses the NEXT_PUBLIC_PERSISTENCE environment variable. When set to 0, the client loads Dexie-based IndexedDB stores from files like lib/persistence/document-access.ts. Server-side code always uses getServerPersistenceProvider to create PostgreSQL-backed stores, with the implementation selected at build or runtime through module resolution.

What is the purpose of separating Asset Store from Asset Byte Store?

The PgAssetStore tracks metadata (filename, MIME type, ownership) in PostgreSQL for fast querying and transactional integrity. The lazyAssetByteStore streams actual bytes from S3, separating hot metadata from cold storage to reduce database size and leverage S3's cost structure for large binary objects.

Can I implement a custom store for a different database backend?

Yes. Any custom store must implement the same TypeScript interfaces exported by the core store files—DocumentStore, RuntimeStore, AssetStore, etc. The omitUndefinedObjectMembers utility in lib/persistence/plain-json.ts handles object canonicalization, so custom stores receive pre-processed JSON-compatible data regardless of upstream transformations.

Where is owner-based data isolation enforced in the persistence layer?

Isolation occurs in OwnerBoundDocumentStore (lib/persistence/owner-bound-document-store.ts) and OwnerMaterials (lib/persistence/owner-materials.ts). These implementations automatically scope all queries to the authenticated owner ID, ensuring multi-tenant deployments maintain strict data boundaries without application-layer filtering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}