# How the OpenMAIC Storage Layer Handles KV Pairs, Documents, and Assets

> Discover how OpenMAIC's storage layer efficiently manages KV pairs, documents, and assets using specialized orthogonal layers for optimized data handling and consistency.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: architecture
- Published: 2026-09-11

---

**OpenMAIC separates persistence into three orthogonal layers—Document Store for heavy course structures, Dexie Metadata DB for relational folder and media data, and KV Store for small user settings—each optimized for specific data shapes with specialized consistency guarantees like epoch-based delete fencing and per-key state machines.**

The OpenMAIC storage layer implements a tiered persistence strategy that treats documents, key-value pairs, and binary assets as distinct concerns requiring different backend optimizations. This architecture, maintained in the THU-MAIC/OpenMAIC repository, ensures that large course documents receive atomic transactional protection while small configuration values benefit from automatic retry and replay semantics.

## The Three-Layer Storage Architecture

OpenMAIC partitions storage responsibilities across three specialized backends:

- **Document Store**: Full course documents including stages, scenes, and outlines, backed by IndexedDB via `@openmaic/storage` (`DocumentStore`)
- **Dexie Metadata DB**: Relational tables for folder membership, media file records, chat sessions, and playback state, managed through Dexie
- **KV Store**: Tiny mutable blobs such as user settings and runtime keys, wrapped by [`kv-persist.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/kv-persist.ts) and backed by `BrowserKVStore` (namespaced localStorage or remote KV)

## Document Store: Atomic Course Persistence

All heavy course data lives in the **document store**, centralized in [`lib/utils/stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/stage-storage.ts). This layer provides atomic, version-guarded writes for course structures while implementing deletion fencing to prevent stale overwrites.

### Atomic and Incremental Saves

The `saveStageData` function in [`lib/utils/stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/stage-storage.ts) (lines 22-62) orchestrates document persistence through `mutateDocument`, which obtains a per-stage document lock and writes under `withRuntimeStorageSharedLock` to ensure consistency between document and runtime chat stores.

For performance, `saveStageDataIncremental` determines which document parts actually changed—scenes, outlines, or stage metadata—and writes only those rows, falling back to full saves only on version errors:

```typescript
import { saveStageDataIncremental } from '@/lib/utils/stage-storage';

const dirty = [{ kind: 'scene', sceneId: editedScene.id }];
await saveStageDataIncremental(stageId, dirty, data, epoch);

```

### Epoch-Based Delete Fencing

Every write receives a `capturedEpoch` value. The `isStageWriteStale(stageId, capturedEpoch)` check (lines 28-30) drops writes captured before a deletion epoch, returning `'stale-dropped'` if a stage was deleted while the write was pending. This prevents restoration of deleted stages from overwriting intentional deletions.

## Dexie Metadata Database

Auxiliary relational data that falls outside the document contract resides in [`lib/utils/database.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/database.ts) (lines 60-107), managed through Dexie over IndexedDB.

### Folder Organization and Media Assets

The metadata database maintains `FolderRecord` and `StageFolderMembership` tables for device-local folder organization, never syncing these server-side. Media assets use `ImageFileRecord`, `AudioFileRecord`, and `MediaFile` rows to store binary blobs and generated IDs.

When stages are deleted, `buildStageAssetReclamationPlan` and `executeStageAssetReclamation` (lines 410-440 in [`stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stage-storage.ts)) identify orphaned assets and reclaim storage.

### Chat Sessions and Playback State

`ChatSessionRecord` stores learner-runtime chat data separately from the document store to keep document schemas small, while playback cursors and legacy migration tables reside in dedicated Dexie tables for fast keyed lookups.

## KV Store: Small Mutable Values

Small, frequently mutated values like user settings and feature flags use the **KV Store**, implemented in [`lib/store/kv-persist.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/store/kv-persist.ts). This wrapper adds a per-key state machine (`KeyState`) that tracks `unhydrated`, `settled`, `unavailable`, and `clearing` phases.

### Per-Key State Machine and Recovery

The wrapper guarantees *read-then-write* ordering by refusing writes issued during hydration until the authoritative read completes. When backends fail, the state transitions to `unavailable`, logs the error via `requestRecovery`, and schedules retry attempts.

If writes are refused while the backend is down, values are stored in `refused` state and automatically replayed after successful reads through `concludeRead`:

```typescript
import { createKVPersistStorage } from '@/lib/store/kv-persist';
import { persist } from 'zustand/middleware';

const useSettings = create(
  persist(
    (set) => ({ theme: 'light', setTheme: (t) => set({ theme: t }) }),
    {
      name: 'settings-storage',
      storage: createKVPersistStorage('account'),
    }
  )
);

```

### Account vs Device Scoping

The KV Store supports `account` scope (synced across devices) and `device` scope (local only). The wrapper enforces that device-scoped stores must use device-safe KV implementations, preventing accidental cross-device leakage of local state.

## Asset Resolution and Cleanup

Stage deletion triggers a cascading cleanup process. The `deleteStageData` function (single-flight per stage) initiates `loadStageAssetInventory` to gather references, then executes `buildStageAssetReclamationPlan` and `executeStageAssetReclamation` to remove binary blobs. This cascade also clears Dexie rows for chat sessions, playback cursors, and legacy tables, ensuring no orphaned data remains.

Loading a stage with its thumbnail demonstrates the separation of document and asset resolution:

```typescript
import { loadStageData, getFirstSlideByStages } from '@/lib/utils/stage-storage';

const stage = await loadStageData(stageId);
const thumbnails = await getFirstSlideByStages([stageId]);
const firstSlide = thumbnails[stageId];

```

Here, `loadStageData` pulls the full document plus chat snapshots, while `getFirstSlideByStages` resolves media references into real URLs through the metadata layer.

## Consistency Guarantees Across Layers

The OpenMAIC storage layer enforces strict consistency through three mechanisms:

- **Write-After-Read Ordering**: The KV wrapper prevents stale snapshots from overwriting authoritative values by refusing writes until hydration completes
- **Epoch-Based Fencing**: Document writes include monotonic epochs; any write captured before a deletion epoch is automatically dropped even if the stage is later restored
- **Atomic Document Transactions**: `mutateDocument` uses Dexie's transaction API, ensuring that document changes roll back entirely if any part fails

## Summary

- OpenMAIC divides storage into three layers: Document Store for course structures, Dexie Metadata DB for relational data, and KV Store for settings
- Document Store in [`lib/utils/stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/stage-storage.ts) provides atomic saves, incremental updates, and epoch-based delete fencing
- Metadata Database in [`lib/utils/database.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/database.ts) handles folders, media assets, and chat sessions separately from documents
- KV Store in [`lib/store/kv-persist.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/store/kv-persist.ts) implements per-key state machines with automatic recovery and refused-write replay
- Asset reclamation ensures complete cleanup of binary blobs and relational rows when stages are deleted via `deleteStageData`

## Frequently Asked Questions

### How does OpenMAIC prevent stale writes during stage deletion?

OpenMAIC implements **epoch-based delete fencing** in [`lib/utils/stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/stage-storage.ts). Each stage maintains a deletion epoch, and every write capture includes a `capturedEpoch` timestamp. The `isStageWriteStale` function checks if the captured epoch predates the deletion; if true, the write is dropped with status `'stale-dropped'`, preventing deleted stages from being accidentally restored by delayed writes.

### What is the difference between the Document Store and the KV Store in OpenMAIC?

The **Document Store** in [`stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stage-storage.ts) handles large, structured course documents (stages, scenes, outlines) using IndexedDB with atomic transactions and versioning, optimized for bulk reads and incremental writes. The **KV Store** in [`kv-persist.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/kv-persist.ts) handles small mutable values like user settings using `BrowserKVStore`, featuring per-key state machines, read-then-write ordering guarantees, and automatic retry logic for transient failures.

### How does the KV store handle backend unavailability?

When the KV backend becomes unavailable, the per-key state machine transitions to the `unavailable` state and logs the error via `requestRecovery`. Writes issued during downtime are stored in `refused` state rather than lost. After the backend recovers and a successful read completes, `concludeRead` automatically replays refused writes, ensuring eventual consistency without user intervention.

### Where are chat sessions stored in OpenMAIC?

Chat sessions reside in the **Dexie Metadata Database** ([`lib/utils/database.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/database.ts)) under the `ChatSessionRecord` table, isolated from the document store to maintain small document schemas. During stage saves, chat data integrates with document writes through `withRuntimeStorageSharedLock` in [`lib/utils/stage-storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/stage-storage.ts), ensuring consistent snapshots across both stores while keeping the document schema lightweight.