How the OpenMAIC Storage Layer Handles KV Pairs, Documents, and Assets

OpenMAIC separates persistence into three orthogonal layers—Document Store for heavy course structures, Dexie Metadata DB for relational folder and media data, and KV Store for small user settings—each optimized for specific data shapes with specialized consistency guarantees like epoch-based delete fencing and per-key state machines.

The OpenMAIC storage layer implements a tiered persistence strategy that treats documents, key-value pairs, and binary assets as distinct concerns requiring different backend optimizations. This architecture, maintained in the THU-MAIC/OpenMAIC repository, ensures that large course documents receive atomic transactional protection while small configuration values benefit from automatic retry and replay semantics.

The Three-Layer Storage Architecture

OpenMAIC partitions storage responsibilities across three specialized backends:

• Document Store: Full course documents including stages, scenes, and outlines, backed by IndexedDB via @openmaic/storage (DocumentStore)
• Dexie Metadata DB: Relational tables for folder membership, media file records, chat sessions, and playback state, managed through Dexie
• KV Store: Tiny mutable blobs such as user settings and runtime keys, wrapped by kv-persist.ts and backed by BrowserKVStore (namespaced localStorage or remote KV)

Document Store: Atomic Course Persistence

All heavy course data lives in the document store, centralized in lib/utils/stage-storage.ts. This layer provides atomic, version-guarded writes for course structures while implementing deletion fencing to prevent stale overwrites.

Atomic and Incremental Saves

The saveStageData function in lib/utils/stage-storage.ts (lines 22-62) orchestrates document persistence through mutateDocument, which obtains a per-stage document lock and writes under withRuntimeStorageSharedLock to ensure consistency between document and runtime chat stores.

For performance, saveStageDataIncremental determines which document parts actually changed—scenes, outlines, or stage metadata—and writes only those rows, falling back to full saves only on version errors:

import { saveStageDataIncremental } from '@/lib/utils/stage-storage';

const dirty = [{ kind: 'scene', sceneId: editedScene.id }];
await saveStageDataIncremental(stageId, dirty, data, epoch);

Epoch-Based Delete Fencing

Every write receives a capturedEpoch value. The isStageWriteStale(stageId, capturedEpoch) check (lines 28-30) drops writes captured before a deletion epoch, returning 'stale-dropped' if a stage was deleted while the write was pending. This prevents restoration of deleted stages from overwriting intentional deletions.

Dexie Metadata Database

Auxiliary relational data that falls outside the document contract resides in lib/utils/database.ts (lines 60-107), managed through Dexie over IndexedDB.

Folder Organization and Media Assets

The metadata database maintains FolderRecord and StageFolderMembership tables for device-local folder organization, never syncing these server-side. Media assets use ImageFileRecord, AudioFileRecord, and MediaFile rows to store binary blobs and generated IDs.

When stages are deleted, buildStageAssetReclamationPlan and executeStageAssetReclamation (lines 410-440 in stage-storage.ts) identify orphaned assets and reclaim storage.

Chat Sessions and Playback State

ChatSessionRecord stores learner-runtime chat data separately from the document store to keep document schemas small, while playback cursors and legacy migration tables reside in dedicated Dexie tables for fast keyed lookups.

KV Store: Small Mutable Values

Small, frequently mutated values like user settings and feature flags use the KV Store, implemented in lib/store/kv-persist.ts. This wrapper adds a per-key state machine (KeyState) that tracks unhydrated, settled, unavailable, and clearing phases.

Per-Key State Machine and Recovery

The wrapper guarantees read-then-write ordering by refusing writes issued during hydration until the authoritative read completes. When backends fail, the state transitions to unavailable, logs the error via requestRecovery, and schedules retry attempts.

If writes are refused while the backend is down, values are stored in refused state and automatically replayed after successful reads through concludeRead:

import { createKVPersistStorage } from '@/lib/store/kv-persist';
import { persist } from 'zustand/middleware';

const useSettings = create(
  persist(
    (set) => ({ theme: 'light', setTheme: (t) => set({ theme: t }) }),
    {
      name: 'settings-storage',
      storage: createKVPersistStorage('account'),
    }
  )
);

Account vs Device Scoping

The KV Store supports account scope (synced across devices) and device scope (local only). The wrapper enforces that device-scoped stores must use device-safe KV implementations, preventing accidental cross-device leakage of local state.

Asset Resolution and Cleanup

Stage deletion triggers a cascading cleanup process. The deleteStageData function (single-flight per stage) initiates loadStageAssetInventory to gather references, then executes buildStageAssetReclamationPlan and executeStageAssetReclamation to remove binary blobs. This cascade also clears Dexie rows for chat sessions, playback cursors, and legacy tables, ensuring no orphaned data remains.

Loading a stage with its thumbnail demonstrates the separation of document and asset resolution:

import { loadStageData, getFirstSlideByStages } from '@/lib/utils/stage-storage';

const stage = await loadStageData(stageId);
const thumbnails = await getFirstSlideByStages([stageId]);
const firstSlide = thumbnails[stageId];

Here, loadStageData pulls the full document plus chat snapshots, while getFirstSlideByStages resolves media references into real URLs through the metadata layer.

Consistency Guarantees Across Layers

The OpenMAIC storage layer enforces strict consistency through three mechanisms:

• Write-After-Read Ordering: The KV wrapper prevents stale snapshots from overwriting authoritative values by refusing writes until hydration completes
• Epoch-Based Fencing: Document writes include monotonic epochs; any write captured before a deletion epoch is automatically dropped even if the stage is later restored
• Atomic Document Transactions: mutateDocument uses Dexie's transaction API, ensuring that document changes roll back entirely if any part fails

Summary

• OpenMAIC divides storage into three layers: Document Store for course structures, Dexie Metadata DB for relational data, and KV Store for settings
• Document Store in lib/utils/stage-storage.ts provides atomic saves, incremental updates, and epoch-based delete fencing
• Metadata Database in lib/utils/database.ts handles folders, media assets, and chat sessions separately from documents
• KV Store in lib/store/kv-persist.ts implements per-key state machines with automatic recovery and refused-write replay
• Asset reclamation ensures complete cleanup of binary blobs and relational rows when stages are deleted via deleteStageData

Frequently Asked Questions

How does OpenMAIC prevent stale writes during stage deletion?

OpenMAIC implements epoch-based delete fencing in lib/utils/stage-storage.ts. Each stage maintains a deletion epoch, and every write capture includes a capturedEpoch timestamp. The isStageWriteStale function checks if the captured epoch predates the deletion; if true, the write is dropped with status 'stale-dropped', preventing deleted stages from being accidentally restored by delayed writes.

What is the difference between the Document Store and the KV Store in OpenMAIC?

The Document Store in stage-storage.ts handles large, structured course documents (stages, scenes, outlines) using IndexedDB with atomic transactions and versioning, optimized for bulk reads and incremental writes. The KV Store in kv-persist.ts handles small mutable values like user settings using BrowserKVStore, featuring per-key state machines, read-then-write ordering guarantees, and automatic retry logic for transient failures.

How does the KV store handle backend unavailability?

When the KV backend becomes unavailable, the per-key state machine transitions to the unavailable state and logs the error via requestRecovery. Writes issued during downtime are stored in refused state rather than lost. After the backend recovers and a successful read completes, concludeRead automatically replays refused writes, ensuring eventual consistency without user intervention.

Where are chat sessions stored in OpenMAIC?

Chat sessions reside in the Dexie Metadata Database (lib/utils/database.ts) under the ChatSessionRecord table, isolated from the document store to maintain small document schemas. During stage saves, chat data integrates with document writes through withRuntimeStorageSharedLock in lib/utils/stage-storage.ts, ensuring consistent snapshots across both stores while keeping the document schema lightweight.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}