# How the Three-Layer Asset Model Works in OpenMAIC Storage

> Learn how the three-layer asset model in OpenMAIC Storage separates metadata, binary storage, and garbage collection for portable references, crash-safe transactions, and deadlock-free reclamation.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: architecture
- Published: 2026-09-10

---

**The `@openmaic/storage` package implements a three-layer architecture that separates asset metadata, binary storage, and garbage collection into isolated concerns, enabling portable document references, crash-safe transactions, and deadlock-free storage reclamation.**

The three-layer asset model serves as the backbone of binary asset management within the OpenMAIC DSL ecosystem. By decoupling stable document references from physical storage locations and maintenance operations, this architecture allows documents to remain backend-agnostic while ensuring data integrity through strict transaction ordering. The separation prevents race conditions during garbage collection and supports seamless migration between diverse storage providers ranging from PostgreSQL to S3 and IndexedDB.

## The Three Layers of Asset Management

### Layer 1: Registry and Metadata Layer

The **Registry** layer maintains stable references and metadata independent of physical storage locations. Defined in `packages/@openmaic/dsl/src/storage.ts`, the `StorageProvider` interface exposes the core methods `put`, `resolve`, and `remove` that clients interact with directly.

When `put` is called, the registry creates a new **`AssetRef`**—a stable, opaque string identifier—and writes a row containing the **`ContentHash`** of the bytes plus **`AssetMeta`** (content type, ownership, reference counts). The `resolve` method looks up this row by the hash and yields a concrete URL for rendering, while `remove` deletes the registry row and flags the underlying bytes for later garbage collection. This layer ensures documents only store portable `AssetRef` values rather than brittle URLs.

### Layer 2: Byte Store Layer

The **Byte Store** holds raw binary payloads (images, audio, video) independently of the registry’s transaction scope. The `AssetByteStore` interface in `packages/@openmaic/storage/src/asset/byte-store.ts` defines the contract with methods `write`, `read`, `delete`, and optional `signReadUrl` for temporary access URLs.

Crucially, bytes are written **before** the metadata row is inserted into the registry. This ordering guarantees crash safety: if the process fails between these two steps, the system loses only the orphaned bytes (which can be re-written), never creating a dangling reference in the registry. Deletion of bytes is deferred until the last registry row referencing them is removed, and only the offline collector performs this deletion to prevent deadlocks.

### Layer 3: Offline Collector Layer

The **Offline Collector** handles garbage collection without interfering with live operations. Implemented in `packages/@openmaic/storage/src/asset/collector.ts`, this layer periodically scans the registry for rows whose reference count has dropped to zero.

Because the byte store operates **offline** and never participates in registry transactions, the collector can safely call `AssetByteStore.delete` to reclaim storage without risking race conditions or self-deadlocks. This offline approach decouples storage maintenance from user-facing request processing, enabling aggressive cleanup even under high load.

## Why Three Layers? Design Benefits

| Benefit | Implementation Detail |
|---------|----------------------|
| **Portability** | Documents store only stable `AssetRef` identifiers. Concrete URLs are resolved at render time via `StorageProvider.resolve`, allowing the same document to move across backends (IndexedDB ↔ S3 ↔ CDN) without mutation. |
| **Atomicity & Crash Safety** | The two-phase write (bytes first, metadata second) ensures that a crash between steps loses only unreferenced bytes, never corrupting the document’s asset reference. |
| **Scalable Garbage Collection** | By isolating the collector from registry transactions, the system performs background byte deletion without locking rows or blocking live asset requests. |

## Working with the Asset Model

The following example demonstrates composing the three layers using PostgreSQL for the registry and S3 for byte storage:

```typescript
import { StorageProvider } from '@openmaic/dsl';
import { createS3ByteStore, createPgRegistry } from '@openmaic/storage';

// 1️⃣ Build the three layers
const byteStore = createS3ByteStore(/* …config… */);
const registry = createPgRegistry({ byteStore });
const storage: StorageProvider = registry; // implements put/resolve/remove

// 2️⃣ Put an asset (binary data → stable ref)
const blob: Blob = new Blob([await fetch('/cat.png').then(r => r.arrayBuffer())], { type: 'image/png' });
const ref = await storage.put(blob, { contentType: 'image/png' });
// `ref` is a stable string that can be stored inside a DSL document

// 3️⃣ Resolve the ref later (e.g. in a renderer)
const url = await storage.resolve(ref);
if (url) {
  // Use the URL in an <img> tag, video src, etc.
  console.log('Resolved URL:', url);
}

// 4️⃣ Remove an asset (registry row is deleted; bytes linger until collector runs)
await storage.remove(ref);
// The offline collector will eventually call `byteStore.delete(hash)` for the orphaned bytes.

```

## Core Source Files and Responsibilities

- **`packages/@openmaic/dsl/src/storage.ts`** – Defines the public `StorageProvider` interface and core types `AssetRef` and `AssetMeta` used by the DSL.
- **`packages/@openmaic/storage/src/asset/byte-store.ts`** – Implements the **Byte Store** interface (`AssetByteStore`) with methods for writing, reading, and deleting binary payloads.
- **`packages/@openmaic/storage/src/asset/types.ts`** – Contains type definitions that tie `ContentHash` to `AssetRef` and expose the registry API contracts.
- **`packages/@openmaic/storage/src/asset/pg.ts`** – PostgreSQL implementation of the **Registry** layer, storing metadata rows and hash links to the Byte Store.
- **`packages/@openmaic/storage/src/asset/collector.ts`** – Houses the offline garbage-collection logic that deletes orphaned bytes after reference counts reach zero.
- **`packages/@openmaic/storage/src/runtime/types.ts`** – Provides runtime utilities such as `ContentHash` used across both registry and byte-store layers.

## Summary

- The **three-layer asset model** separates concerns into Registry (metadata), Byte Store (binary data), and Offline Collector (garbage collection).
- **Registry layer** stores stable `AssetRef` identifiers and metadata in `packages/@openmaic/dsl/src/storage.ts`, enabling document portability.
- **Byte Store layer** manages raw payloads via `AssetByteStore` in `packages/@openmaic/storage/src/asset/byte-store.ts`, with write-before-metadata ordering for crash safety.
- **Offline Collector** in `packages/@openmaic/storage/src/asset/collector.ts` reclaims storage without blocking live transactions or causing deadlocks.
- This architecture supports mixing storage backends (e.g., PostgreSQL for metadata, S3 for bytes) while maintaining atomicity and scalable cleanup.

## Frequently Asked Questions

### What is the purpose of separating the Registry from the Byte Store?

Separating these layers allows the Registry to remain lightweight and transactional while the Byte Store handles large binary payloads independently. This isolation prevents database transactions from locking during slow upload operations and enables the Byte Store to reside on an entirely different backend (such as S3 or local filesystem) from the Registry (such as PostgreSQL).

### How does the three-layer model prevent data loss during application crashes?

The system writes bytes to the Byte Store **before** inserting the metadata row into the Registry. If a crash occurs between these operations, the result is unreferenced bytes (which the Offline Collector will eventually clean up) rather than a document containing a reference to non-existent data. This creates a fail-safe where data can be re-uploaded but documents never point to missing assets.

### When does the Offline Collector actually delete asset bytes?

The collector deletes bytes only after the Registry confirms that no rows reference the corresponding `ContentHash` (reference count equals zero). This deletion happens asynchronously in `packages/@openmaic/storage/src/asset/collector.ts`, outside of any user request transaction, ensuring that removal of large files does not block or deadlock active document editing operations.

### Can I use different storage backends for the Registry and Byte Store layers?

Yes. The architecture explicitly supports mixing backends—for example, using `createPgRegistry` for PostgreSQL metadata storage and `createS3ByteStore` for S3 binary storage. Because the `StorageProvider` interface abstracts the relationship between layers, you can combine any compatible Registry implementation with any `AssetByteStore` implementation without changing document or application code.