# Pluggable Persistence in OpenMAIC: Storage Architecture and Backend Options

> Explore OpenMAIC's pluggable persistence architecture. Discover storage primitives and backend options like Browser, HTTP Client, and PostgreSQL for flexible data management.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: architecture
- Published: 2026-08-30

---

**OpenMAIC implements a four-tier pluggable persistence architecture that separates storage primitives (KV, Document, Runtime, and Asset) from their physical backends, enabling seamless swapping between Browser (IndexedDB/localStorage), HTTP Client, and PostgreSQL implementations without changing application logic.**

OpenMAIC's data layer is designed around a pluggable persistence stack that decouples *what* data is stored from *where* it resides. According to the THU-MAIC/OpenMAIC source code, this architecture allows the same application to run entirely client-side, connect to a remote API, or deploy as a self-hosted PostgreSQL instance by injecting different storage implementations through constructor parameters.

## The Four Persistence Primitives

OpenMAIC organizes persistence into four primitive categories, each representing a distinct data access pattern. Every primitive exposes a consistent TypeScript interface while allowing backend-specific implementations to handle the actual I/O.

### KVStore for Scoped Key-Value Data

The **KVStore** primitive manages small scoped values at the device or account level that exist outside the DSL-defined document structure. These values persist user preferences, authentication tokens, and ephemeral state. The interface supports get/set operations with explicit scoping to prevent cross-contamination between anonymous device sessions and authenticated accounts.

### DocumentStore for DSL Documents

**DocumentStore** handles normalized persistence of the DSL document aggregate, including stages, scenes, outlines, and their hierarchical relationships. This store manages the primary content model that defines learning experiences, ensuring atomic updates to complex document trees while maintaining referential integrity across the graph.

### RuntimeStore for Session Data

**RuntimeStore** provides append-only storage for learner session data such as chat logs, quiz attempts, and playback facts. Unlike the DocumentStore which manages relatively static content, the RuntimeStore handles high-velocity write streams that grow continuously throughout a learning session.

### AssetStore for Media and Binary Data

**AssetStore** manages media asset metadata and byte storage through a two-layer architecture. The registry layer tracks asset IDs, MIME types, and content hashes, while the **pluggable byte layer** (defined in [`lib/persistence/asset-byte-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/asset-byte-store.ts)) handles the actual blob storage. This separation allows the metadata to live in PostgreSQL while the bytes reside in S3, local filesystem, or database columns depending on deployment constraints.

## Backend Implementation Options

Each primitive supports three distinct backend tiers, selectable via dependency injection at runtime.

### Browser Backends (Client-Side Only)

The browser implementations leverage web storage APIs for offline-capable deployments:

- **BrowserKVStore**: Wraps `localStorage` for simple key-value persistence
- **BrowserDocumentStore**: Uses IndexedDB object store `maic-documents` for结构化 content
- **BrowserRuntimeStore**: Writes to IndexedDB `maic-runtime` object store for session data
- **BrowserAssetStore**: Maintains an IndexedDB registry with Blob storage and object URL generation

These implementations reside in `packages/@openmaic/storage/src/` with specific files like [`kv/browser.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/kv/browser.ts) and [`document/browser.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/document/browser.ts).

### HTTP Client Backends (Serverless/Remote)

HTTP client implementations act as thin wrappers around REST endpoints, enabling stateless deployments where the browser delegates persistence to a remote server:

- **HttpKVStore**: Communicates with `/api/persistence/kv` endpoints
- **HttpDocumentStore**: Proxies document operations to the reference server's document API
- **HttpRuntimeStore**: Streams runtime records to `/api/persistence/runtime`
- **HttpAssetStore**: Manages asset metadata via the HTTP contract defined in `packages/@openmaic/storage/docs/asset-http-contract.md`

### PostgreSQL Server Backends (Self-Hosted)

The server implementations provide ACID-compliant persistence for production deployments:

- **PgKVStore**: Stores scoped key-value pairs in PostgreSQL via the `PgRuntimeStore` interface
- **PgDocumentStore**: Normalizes document aggregates into relational tables
- **PgRuntimeStore**: Efficiently handles append-only inserts for high-volume session data
- **PgAssetStore**: Stores asset metadata in PostgreSQL with configurable byte storage (column, external object store, or content-addressed storage)

These are instantiated through `getServerPersistenceProvider` in [`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts), which memoizes a single provider instance per connection string to prevent connection pool exhaustion.

## How Pluggability Works

### Seam Injection Architecture

All backend implementations accept their low-level storage mechanism through constructor parameters rather than hard-coding dependencies. For example, `BrowserKVStore` receives an `IDBFactory` or `localStorage` reference, while `HttpKVStore` receives a `fetch` implementation and base URL. This seam injection allows the same high-level business logic to operate across browser, Node.js, and test environments with mocked storage.

### Canonicalization Layer

Before persisting to JSON, objects pass through `omitUndefinedObjectMembers` in [`lib/persistence/plain-json.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/plain-json.ts). This utility strips `undefined` values to ensure lossless round-tripping across the JSON boundary, preventing type mismatches when hydrating data from different backends.

### Scope Awareness

Both KV and Document primitives carry a scope discriminator (`device` vs `account`). The HTTP KV contract enforces `account`-only operations at the server boundary, while the client injects a local `BrowserKVStore` for `device`-scoped values. This ensures sensitive account data syncs across devices while ephemeral device state remains local.

### Asset Byte Layer Abstraction

The asset system separates metadata storage from byte storage. The `PgAssetStore` persists metadata to PostgreSQL but delegates byte operations to a pluggable byte store configured via `lazyAssetByteStore` in [`lib/persistence/asset-byte-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/asset-byte-store.ts). This enables configurations where metadata lives in PostgreSQL while bytes stream to S3-compatible object storage or remain inline as bytea columns.

## Implementation Examples

### Configuring Browser-Only Persistence

For client-side deployments using IndexedDB and localStorage:

```typescript
import { BrowserKVStore } from '@openmaic/storage/kv/browser';
import { BrowserDocumentStore } from '@openmaic/storage/document/browser';
import { BrowserRuntimeStore } from '@openmaic/storage/runtime/browser';
import { BrowserAssetStore } from '@openmaic/storage/asset/browser';

const kvStore = new BrowserKVStore(localStorage);
const documentStore = new BrowserDocumentStore(indexedDB);
const runtimeStore = new BrowserRuntimeStore(indexedDB);
const assetStore = new BrowserAssetStore(indexedDB);

// Store user preference
await kvStore.set('userTheme', 'dark');

// Persist document structure
await documentStore.putStage({
  id: 'stage_01',
  title: 'Introduction',
  scenes: [{ id: 'scene_01', type: 'slide', content: 'Welcome' }],
});

// Append runtime record
await runtimeStore.appendRecord('stage_01', {
  learnerKey: 'anonymous',
  kind: 'chat',
  payload: { text: 'Hello world' }
});

```

### Configuring HTTP Client Persistence

For serverless or thin-client deployments delegating to a remote API:

```typescript
import { HttpKVStore } from '@openmaic/storage/kv/http';
import { HttpDocumentStore } from '@openmaic/storage/document/http';
import { HttpRuntimeStore } from '@openmaic/storage/runtime/http';
import { HttpAssetStore } from '@openmaic/storage/asset/http';

const apiBase = 'https://api.example.com/persistence';
const fetchImpl = fetch; // or node-fetch in serverless environments

const kvStore = new HttpKVStore(fetchImpl, `${apiBase}/kv`);
const documentStore = new HttpDocumentStore(fetchImpl, `${apiBase}/documents`);
const runtimeStore = new HttpRuntimeStore(fetchImpl, `${apiBase}/runtime`);
const assetStore = new HttpAssetStore(fetchImpl, `${apiBase}/assets`);

// Account-scoped KV (device scope handled separately)
await kvStore.set('account:theme', 'light');

// Remote document persistence
await documentStore.putStage({ id: 'stage_02', /* ... */ });

// Stream runtime data to server
await runtimeStore.appendRecord('stage_02', {
  learnerKey: 'user42',
  kind: 'quizAttempt',
  payload: { score: 8 }
});

```

### Configuring PostgreSQL Server Persistence

For full-stack or self-hosted deployments with direct database access:

```typescript
import { getServerPersistenceProvider } from '@/lib/persistence/server-provider';

async function initializePersistence() {
  const connectionString = process.env.DATABASE_URL;
  const provider = await getServerPersistenceProvider(connectionString);
  
  const { kvStore, documentStore, runtimeStore, assetStore } = provider;
  
  // Cross-device synchronized preferences
  await kvStore.set('account:language', 'en');
  
  // Complex document with scenes
  await documentStore.putStage({
    id: 'stage_99',
    title: 'Advanced Concepts',
    scenes: [{ id: 'scene_99', type: 'interactive', content: 'Deep dive' }],
  });
  
  // Asset with pluggable byte storage
  const assetId = await assetStore.putMetadata({
    mime: 'image/png',
    contentHash: 'sha256:abc123...'
  });
  
  // Upload bytes to configured byte layer (S3, column, or filesystem)
  const byteStore = await import('@/lib/persistence/asset-byte-store')
    .then(m => m.lazyAssetByteStore);
  await byteStore(connectionString, provider).upload(assetId, imageBlob);
}

```

## Summary

- OpenMAIC provides **four persistence primitives** (KVStore, DocumentStore, RuntimeStore, AssetStore) that abstract storage patterns from physical implementations.
- Each primitive supports **three backend tiers**: Browser (IndexedDB/localStorage), HTTP Client (REST API), and PostgreSQL Server (ACID-compliant relational storage).
- **Seam injection** via constructor parameters enables swapping backends without modifying application business logic.
- The **AssetStore** uniquely separates metadata storage from byte storage, allowing flexible configurations like PostgreSQL metadata with S3-compatible object storage.
- **Scope awareness** (`device` vs `account`) ensures proper data isolation, with HTTP clients automatically delegating device scope to local browser storage while syncing account scope to the server.
- **Global provider caching** via `getServerPersistenceProvider` prevents redundant PostgreSQL connection pool creation in server environments.

## Frequently Asked Questions

### What determines which persistence backend OpenMAIC uses at runtime?

The backend selection depends on which store implementations are instantiated and injected into the application layer. Client-side builds import from `@openmaic/storage/*/browser`, serverless functions use `*/http` clients, and self-hosted servers import directly from [`lib/persistence/server-provider.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/server-provider.ts) to receive PostgreSQL implementations. The interface contracts remain identical across all three tiers.

### How does OpenMAIC handle large binary assets without bloating the primary database?

The AssetStore architecture separates metadata from bytes. While `PgAssetStore` stores asset metadata (ID, MIME type, hash) in PostgreSQL, the actual bytes flow through the pluggable byte layer defined in [`lib/persistence/asset-byte-store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/asset-byte-store.ts). This layer can be configured to store blobs as PostgreSQL bytea columns, offload to external S3-compatible object storage, or use content-addressed storage systems depending on deployment scale and latency requirements.

### Can I mix different backend types within the same application?

Yes. OpenMAIC's architecture supports hybrid persistence configurations common in progressive web applications. For example, you can configure `BrowserKVStore` for device-scoped settings while using `HttpKVStore` for account-scoped data, or cache documents locally with `BrowserDocumentStore` while syncing to `PgDocumentStore` when connectivity is available. The scope awareness system automatically routes data to the appropriate backend based on the `device` or `account` namespace.

### What serialization format does OpenMAIC use for cross-backend compatibility?

All JSON persistence flows through `omitUndefinedObjectMembers` in [`lib/persistence/plain-json.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/persistence/plain-json.ts), which strips `undefined` values before serialization. This canonicalization ensures that objects round-trip identically whether stored in IndexedDB, transmitted over HTTP, or persisted to PostgreSQL JSONB columns, preventing undefined vs null mismatches across JavaScript runtime boundaries.