Pluggable Persistence in OpenMAIC: Storage Architecture and Backend Options

OpenMAIC implements a four-tier pluggable persistence architecture that separates storage primitives (KV, Document, Runtime, and Asset) from their physical backends, enabling seamless swapping between Browser (IndexedDB/localStorage), HTTP Client, and PostgreSQL implementations without changing application logic.

OpenMAIC's data layer is designed around a pluggable persistence stack that decouples what data is stored from where it resides. According to the THU-MAIC/OpenMAIC source code, this architecture allows the same application to run entirely client-side, connect to a remote API, or deploy as a self-hosted PostgreSQL instance by injecting different storage implementations through constructor parameters.

The Four Persistence Primitives

OpenMAIC organizes persistence into four primitive categories, each representing a distinct data access pattern. Every primitive exposes a consistent TypeScript interface while allowing backend-specific implementations to handle the actual I/O.

KVStore for Scoped Key-Value Data

The KVStore primitive manages small scoped values at the device or account level that exist outside the DSL-defined document structure. These values persist user preferences, authentication tokens, and ephemeral state. The interface supports get/set operations with explicit scoping to prevent cross-contamination between anonymous device sessions and authenticated accounts.

DocumentStore for DSL Documents

DocumentStore handles normalized persistence of the DSL document aggregate, including stages, scenes, outlines, and their hierarchical relationships. This store manages the primary content model that defines learning experiences, ensuring atomic updates to complex document trees while maintaining referential integrity across the graph.

RuntimeStore for Session Data

RuntimeStore provides append-only storage for learner session data such as chat logs, quiz attempts, and playback facts. Unlike the DocumentStore which manages relatively static content, the RuntimeStore handles high-velocity write streams that grow continuously throughout a learning session.

AssetStore for Media and Binary Data

AssetStore manages media asset metadata and byte storage through a two-layer architecture. The registry layer tracks asset IDs, MIME types, and content hashes, while the pluggable byte layer (defined in lib/persistence/asset-byte-store.ts) handles the actual blob storage. This separation allows the metadata to live in PostgreSQL while the bytes reside in S3, local filesystem, or database columns depending on deployment constraints.

Backend Implementation Options

Each primitive supports three distinct backend tiers, selectable via dependency injection at runtime.

Browser Backends (Client-Side Only)

The browser implementations leverage web storage APIs for offline-capable deployments:

• BrowserKVStore: Wraps localStorage for simple key-value persistence
• BrowserDocumentStore: Uses IndexedDB object store maic-documents for结构化 content
• BrowserRuntimeStore: Writes to IndexedDB maic-runtime object store for session data
• BrowserAssetStore: Maintains an IndexedDB registry with Blob storage and object URL generation

These implementations reside in packages/@openmaic/storage/src/ with specific files like kv/browser.ts and document/browser.ts.

HTTP Client Backends (Serverless/Remote)

HTTP client implementations act as thin wrappers around REST endpoints, enabling stateless deployments where the browser delegates persistence to a remote server:

• HttpKVStore: Communicates with /api/persistence/kv endpoints
• HttpDocumentStore: Proxies document operations to the reference server's document API
• HttpRuntimeStore: Streams runtime records to /api/persistence/runtime
• HttpAssetStore: Manages asset metadata via the HTTP contract defined in packages/@openmaic/storage/docs/asset-http-contract.md

PostgreSQL Server Backends (Self-Hosted)

The server implementations provide ACID-compliant persistence for production deployments:

• PgKVStore: Stores scoped key-value pairs in PostgreSQL via the PgRuntimeStore interface
• PgDocumentStore: Normalizes document aggregates into relational tables
• PgRuntimeStore: Efficiently handles append-only inserts for high-volume session data
• PgAssetStore: Stores asset metadata in PostgreSQL with configurable byte storage (column, external object store, or content-addressed storage)

These are instantiated through getServerPersistenceProvider in lib/persistence/server-provider.ts, which memoizes a single provider instance per connection string to prevent connection pool exhaustion.

How Pluggability Works

Seam Injection Architecture

All backend implementations accept their low-level storage mechanism through constructor parameters rather than hard-coding dependencies. For example, BrowserKVStore receives an IDBFactory or localStorage reference, while HttpKVStore receives a fetch implementation and base URL. This seam injection allows the same high-level business logic to operate across browser, Node.js, and test environments with mocked storage.

Canonicalization Layer

Before persisting to JSON, objects pass through omitUndefinedObjectMembers in lib/persistence/plain-json.ts. This utility strips undefined values to ensure lossless round-tripping across the JSON boundary, preventing type mismatches when hydrating data from different backends.

Scope Awareness

Both KV and Document primitives carry a scope discriminator (device vs account). The HTTP KV contract enforces account-only operations at the server boundary, while the client injects a local BrowserKVStore for device-scoped values. This ensures sensitive account data syncs across devices while ephemeral device state remains local.

Asset Byte Layer Abstraction

The asset system separates metadata storage from byte storage. The PgAssetStore persists metadata to PostgreSQL but delegates byte operations to a pluggable byte store configured via lazyAssetByteStore in lib/persistence/asset-byte-store.ts. This enables configurations where metadata lives in PostgreSQL while bytes stream to S3-compatible object storage or remain inline as bytea columns.

Implementation Examples

Configuring Browser-Only Persistence

For client-side deployments using IndexedDB and localStorage:

import { BrowserKVStore } from '@openmaic/storage/kv/browser';
import { BrowserDocumentStore } from '@openmaic/storage/document/browser';
import { BrowserRuntimeStore } from '@openmaic/storage/runtime/browser';
import { BrowserAssetStore } from '@openmaic/storage/asset/browser';

const kvStore = new BrowserKVStore(localStorage);
const documentStore = new BrowserDocumentStore(indexedDB);
const runtimeStore = new BrowserRuntimeStore(indexedDB);
const assetStore = new BrowserAssetStore(indexedDB);

// Store user preference
await kvStore.set('userTheme', 'dark');

// Persist document structure
await documentStore.putStage({
  id: 'stage_01',
  title: 'Introduction',
  scenes: [{ id: 'scene_01', type: 'slide', content: 'Welcome' }],
});

// Append runtime record
await runtimeStore.appendRecord('stage_01', {
  learnerKey: 'anonymous',
  kind: 'chat',
  payload: { text: 'Hello world' }
});

Configuring HTTP Client Persistence

For serverless or thin-client deployments delegating to a remote API:

import { HttpKVStore } from '@openmaic/storage/kv/http';
import { HttpDocumentStore } from '@openmaic/storage/document/http';
import { HttpRuntimeStore } from '@openmaic/storage/runtime/http';
import { HttpAssetStore } from '@openmaic/storage/asset/http';

const apiBase = 'https://api.example.com/persistence';
const fetchImpl = fetch; // or node-fetch in serverless environments

const kvStore = new HttpKVStore(fetchImpl, `${apiBase}/kv`);
const documentStore = new HttpDocumentStore(fetchImpl, `${apiBase}/documents`);
const runtimeStore = new HttpRuntimeStore(fetchImpl, `${apiBase}/runtime`);
const assetStore = new HttpAssetStore(fetchImpl, `${apiBase}/assets`);

// Account-scoped KV (device scope handled separately)
await kvStore.set('account:theme', 'light');

// Remote document persistence
await documentStore.putStage({ id: 'stage_02', /* ... */ });

// Stream runtime data to server
await runtimeStore.appendRecord('stage_02', {
  learnerKey: 'user42',
  kind: 'quizAttempt',
  payload: { score: 8 }
});

Configuring PostgreSQL Server Persistence

For full-stack or self-hosted deployments with direct database access:

import { getServerPersistenceProvider } from '@/lib/persistence/server-provider';

async function initializePersistence() {
  const connectionString = process.env.DATABASE_URL;
  const provider = await getServerPersistenceProvider(connectionString);
  
  const { kvStore, documentStore, runtimeStore, assetStore } = provider;
  
  // Cross-device synchronized preferences
  await kvStore.set('account:language', 'en');
  
  // Complex document with scenes
  await documentStore.putStage({
    id: 'stage_99',
    title: 'Advanced Concepts',
    scenes: [{ id: 'scene_99', type: 'interactive', content: 'Deep dive' }],
  });
  
  // Asset with pluggable byte storage
  const assetId = await assetStore.putMetadata({
    mime: 'image/png',
    contentHash: 'sha256:abc123...'
  });
  
  // Upload bytes to configured byte layer (S3, column, or filesystem)
  const byteStore = await import('@/lib/persistence/asset-byte-store')
    .then(m => m.lazyAssetByteStore);
  await byteStore(connectionString, provider).upload(assetId, imageBlob);
}

Summary

• OpenMAIC provides four persistence primitives (KVStore, DocumentStore, RuntimeStore, AssetStore) that abstract storage patterns from physical implementations.
• Each primitive supports three backend tiers: Browser (IndexedDB/localStorage), HTTP Client (REST API), and PostgreSQL Server (ACID-compliant relational storage).
• Seam injection via constructor parameters enables swapping backends without modifying application business logic.
• The AssetStore uniquely separates metadata storage from byte storage, allowing flexible configurations like PostgreSQL metadata with S3-compatible object storage.
• Scope awareness (device vs account) ensures proper data isolation, with HTTP clients automatically delegating device scope to local browser storage while syncing account scope to the server.
• Global provider caching via getServerPersistenceProvider prevents redundant PostgreSQL connection pool creation in server environments.

Frequently Asked Questions

What determines which persistence backend OpenMAIC uses at runtime?

The backend selection depends on which store implementations are instantiated and injected into the application layer. Client-side builds import from @openmaic/storage/*/browser, serverless functions use */http clients, and self-hosted servers import directly from lib/persistence/server-provider.ts to receive PostgreSQL implementations. The interface contracts remain identical across all three tiers.

How does OpenMAIC handle large binary assets without bloating the primary database?

The AssetStore architecture separates metadata from bytes. While PgAssetStore stores asset metadata (ID, MIME type, hash) in PostgreSQL, the actual bytes flow through the pluggable byte layer defined in lib/persistence/asset-byte-store.ts. This layer can be configured to store blobs as PostgreSQL bytea columns, offload to external S3-compatible object storage, or use content-addressed storage systems depending on deployment scale and latency requirements.

Can I mix different backend types within the same application?

Yes. OpenMAIC's architecture supports hybrid persistence configurations common in progressive web applications. For example, you can configure BrowserKVStore for device-scoped settings while using HttpKVStore for account-scoped data, or cache documents locally with BrowserDocumentStore while syncing to PgDocumentStore when connectivity is available. The scope awareness system automatically routes data to the appropriate backend based on the device or account namespace.

What serialization format does OpenMAIC use for cross-backend compatibility?

All JSON persistence flows through omitUndefinedObjectMembers in lib/persistence/plain-json.ts, which strips undefined values before serialization. This canonicalization ensures that objects round-trip identically whether stored in IndexedDB, transmitted over HTTP, or persisted to PostgreSQL JSONB columns, preventing undefined vs null mismatches across JavaScript runtime boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}