How the Three-Layer Asset Model Works in OpenMAIC Storage

The @openmaic/storage package implements a three-layer architecture that separates asset metadata, binary storage, and garbage collection into isolated concerns, enabling portable document references, crash-safe transactions, and deadlock-free storage reclamation.

The three-layer asset model serves as the backbone of binary asset management within the OpenMAIC DSL ecosystem. By decoupling stable document references from physical storage locations and maintenance operations, this architecture allows documents to remain backend-agnostic while ensuring data integrity through strict transaction ordering. The separation prevents race conditions during garbage collection and supports seamless migration between diverse storage providers ranging from PostgreSQL to S3 and IndexedDB.

The Three Layers of Asset Management

Layer 1: Registry and Metadata Layer

The Registry layer maintains stable references and metadata independent of physical storage locations. Defined in packages/@openmaic/dsl/src/storage.ts, the StorageProvider interface exposes the core methods put, resolve, and remove that clients interact with directly.

When put is called, the registry creates a new AssetRef—a stable, opaque string identifier—and writes a row containing the ContentHash of the bytes plus AssetMeta (content type, ownership, reference counts). The resolve method looks up this row by the hash and yields a concrete URL for rendering, while remove deletes the registry row and flags the underlying bytes for later garbage collection. This layer ensures documents only store portable AssetRef values rather than brittle URLs.

Layer 2: Byte Store Layer

The Byte Store holds raw binary payloads (images, audio, video) independently of the registry’s transaction scope. The AssetByteStore interface in packages/@openmaic/storage/src/asset/byte-store.ts defines the contract with methods write, read, delete, and optional signReadUrl for temporary access URLs.

Crucially, bytes are written before the metadata row is inserted into the registry. This ordering guarantees crash safety: if the process fails between these two steps, the system loses only the orphaned bytes (which can be re-written), never creating a dangling reference in the registry. Deletion of bytes is deferred until the last registry row referencing them is removed, and only the offline collector performs this deletion to prevent deadlocks.

Layer 3: Offline Collector Layer

The Offline Collector handles garbage collection without interfering with live operations. Implemented in packages/@openmaic/storage/src/asset/collector.ts, this layer periodically scans the registry for rows whose reference count has dropped to zero.

Because the byte store operates offline and never participates in registry transactions, the collector can safely call AssetByteStore.delete to reclaim storage without risking race conditions or self-deadlocks. This offline approach decouples storage maintenance from user-facing request processing, enabling aggressive cleanup even under high load.

Why Three Layers? Design Benefits

Benefit	Implementation Detail
Portability	Documents store only stable AssetRef identifiers. Concrete URLs are resolved at render time via StorageProvider.resolve, allowing the same document to move across backends (IndexedDB ↔ S3 ↔ CDN) without mutation.
Atomicity & Crash Safety	The two-phase write (bytes first, metadata second) ensures that a crash between steps loses only unreferenced bytes, never corrupting the document’s asset reference.
Scalable Garbage Collection	By isolating the collector from registry transactions, the system performs background byte deletion without locking rows or blocking live asset requests.

Working with the Asset Model

The following example demonstrates composing the three layers using PostgreSQL for the registry and S3 for byte storage:

import { StorageProvider } from '@openmaic/dsl';
import { createS3ByteStore, createPgRegistry } from '@openmaic/storage';

// 1️⃣ Build the three layers
const byteStore = createS3ByteStore(/* …config… */);
const registry = createPgRegistry({ byteStore });
const storage: StorageProvider = registry; // implements put/resolve/remove

// 2️⃣ Put an asset (binary data → stable ref)
const blob: Blob = new Blob([await fetch('/cat.png').then(r => r.arrayBuffer())], { type: 'image/png' });
const ref = await storage.put(blob, { contentType: 'image/png' });
// `ref` is a stable string that can be stored inside a DSL document

// 3️⃣ Resolve the ref later (e.g. in a renderer)
const url = await storage.resolve(ref);
if (url) {
  // Use the URL in an <img> tag, video src, etc.
  console.log('Resolved URL:', url);
}

// 4️⃣ Remove an asset (registry row is deleted; bytes linger until collector runs)
await storage.remove(ref);
// The offline collector will eventually call `byteStore.delete(hash)` for the orphaned bytes.

Core Source Files and Responsibilities

• packages/@openmaic/dsl/src/storage.ts – Defines the public StorageProvider interface and core types AssetRef and AssetMeta used by the DSL.
• packages/@openmaic/storage/src/asset/byte-store.ts – Implements the Byte Store interface (AssetByteStore) with methods for writing, reading, and deleting binary payloads.
• packages/@openmaic/storage/src/asset/types.ts – Contains type definitions that tie ContentHash to AssetRef and expose the registry API contracts.
• packages/@openmaic/storage/src/asset/pg.ts – PostgreSQL implementation of the Registry layer, storing metadata rows and hash links to the Byte Store.
• packages/@openmaic/storage/src/asset/collector.ts – Houses the offline garbage-collection logic that deletes orphaned bytes after reference counts reach zero.
• packages/@openmaic/storage/src/runtime/types.ts – Provides runtime utilities such as ContentHash used across both registry and byte-store layers.

Summary

• The three-layer asset model separates concerns into Registry (metadata), Byte Store (binary data), and Offline Collector (garbage collection).
• Registry layer stores stable AssetRef identifiers and metadata in packages/@openmaic/dsl/src/storage.ts, enabling document portability.
• Byte Store layer manages raw payloads via AssetByteStore in packages/@openmaic/storage/src/asset/byte-store.ts, with write-before-metadata ordering for crash safety.
• Offline Collector in packages/@openmaic/storage/src/asset/collector.ts reclaims storage without blocking live transactions or causing deadlocks.
• This architecture supports mixing storage backends (e.g., PostgreSQL for metadata, S3 for bytes) while maintaining atomicity and scalable cleanup.

Frequently Asked Questions

What is the purpose of separating the Registry from the Byte Store?

Separating these layers allows the Registry to remain lightweight and transactional while the Byte Store handles large binary payloads independently. This isolation prevents database transactions from locking during slow upload operations and enables the Byte Store to reside on an entirely different backend (such as S3 or local filesystem) from the Registry (such as PostgreSQL).

How does the three-layer model prevent data loss during application crashes?

The system writes bytes to the Byte Store before inserting the metadata row into the Registry. If a crash occurs between these operations, the result is unreferenced bytes (which the Offline Collector will eventually clean up) rather than a document containing a reference to non-existent data. This creates a fail-safe where data can be re-uploaded but documents never point to missing assets.

When does the Offline Collector actually delete asset bytes?

The collector deletes bytes only after the Registry confirms that no rows reference the corresponding ContentHash (reference count equals zero). This deletion happens asynchronously in packages/@openmaic/storage/src/asset/collector.ts, outside of any user request transaction, ensuring that removal of large files does not block or deadlock active document editing operations.

Can I use different storage backends for the Registry and Byte Store layers?

Yes. The architecture explicitly supports mixing backends—for example, using createPgRegistry for PostgreSQL metadata storage and createS3ByteStore for S3 binary storage. Because the StorageProvider interface abstracts the relationship between layers, you can combine any compatible Registry implementation with any AssetByteStore implementation without changing document or application code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}