How to Configure Asset Storage in OpenMAIC: PostgreSQL vs S3 Setup Guide

OpenMAIC determines where binary assets are stored at runtime based on the ASSET_S3_BUCKET environment variable, defaulting to PostgreSQL for local deployments or routing to Amazon S3 when a valid bucket name is configured.

Asset storage configuration in OpenMAIC relies on a pluggable asset byte store architecture that keeps the persistence layer agnostic to underlying storage mechanics. The system supports two distinct backends— PostgreSQL for simplicity or Amazon S3 for scalability—selected automatically without code changes based on environment variables.

Runtime Storage Selection Logic

The storage backend is resolved in lib/persistence/asset-byte-store.ts through a conditional check on process.env.ASSET_S3_BUCKET. This design allows the same deployment artifact to run in development against a local database or in production against object storage.

PostgreSQL Default Storage

When ASSET_S3_BUCKET is unset or empty, OpenMAIC stores asset bytes directly in the application database using PgAssetByteStore. The lazyAssetByteStore function returns a PgForwardedByteStore instance that wraps PostgreSQL connections and exposes transaction-pinned methods (writeWith, readWith, deleteWith). This ensures that asset operations participate in database transactions, maintaining ACID compliance with metadata records.

Amazon S3 Scalable Storage

When ASSET_S3_BUCKET contains a valid bucket name, the system dynamically imports @openmaic/storage/asset/s3-bytes via loadS3AssetByteStore and instantiates S3AssetByteStore. This implementation sets the flag writesOutsideRegistryDatabase: true because S3 writes cannot participate in PostgreSQL transactions, requiring the persistence layer to handle potential inconsistencies explicitly.

Core Configuration in asset-byte-store.ts

The lib/persistence/asset-byte-store.ts file contains three critical functions that govern asset storage configuration.

Bucket Name Validation

The configuredS3Bucket(value) function sanitizes and validates the environment variable before use. It trims whitespace, enforces AWS naming constraints (3–63 characters, allowed character set, reserved prefix/suffix restrictions), and throws descriptive errors for malformed bucket names. This validation prevents runtime failures from invalid configuration strings.

Lazy Initialization Pattern

The lazyAssetByteStore(bucketValue, queryable) factory returns an AssetByteStore that defers actual construction until the first asset request. This laziness ensures that a misconfigured S3 bucket does not crash the server at startup—only asset-related endpoints fail, leaving other application functions operational.

import { lazyAssetByteStore } from '@/lib/persistence/asset-byte-store';
import { getPgQueryable } from '@/lib/persistence/pg';

const queryable = await getPgQueryable();
export const assetByteStore = lazyAssetByteStore(
  process.env.ASSET_S3_BUCKET, 
  queryable
);

// First invocation triggers backend selection
const assetId = await assetByteStore.write(fileHash, fileBuffer);

Transaction Boundary Handling

PostgreSQL storage supports transactional asset operations through writeWith, readWith, and deleteWith methods. S3 storage lacks these methods since object storage operations cannot roll back with database transactions, requiring application-level compensation logic in lib/persistence/server-provider.ts.

Client Egress Configuration

Beyond storage location, OpenMAIC controls how assets reach clients via the ASSET_BYTE_EGRESS environment variable. The routing logic resides in lib/persistence/server-provider.ts and the behavior is validated in tests/persistence/route.test.ts.

When set to redirect, the server generates pre-signed S3 URLs and returns HTTP 302 responses, offloading bandwidth to AWS. Any other value (or when unset) triggers direct streaming, where OpenMAIC reads bytes from the configured store and pipes them through the application server.

Production Configuration Examples

Configure your deployment environment variables to select the appropriate storage backend:

# PostgreSQL-only deployment (development/small scale)

ASSET_S3_BUCKET=

# S3-backed production deployment

ASSET_S3_BUCKET=acme-openmaic-assets-prod
ASSET_BYTE_EGRESS=redirect  # Optional: use signed URLs for downloads

AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=xxx
AWS_SECRET_ACCESS_KEY=yyy

Implementation in application bootstrap:

// lib/persistence/asset-init.ts
import { lazyAssetByteStore } from './asset-byte-store';
import { getQueryable } from './pg';

export const store = lazyAssetByteStore(
  process.env.ASSET_S3_BUCKET,
  getQueryable()
);

Summary

• Environment-driven selection: Set ASSET_S3_BUCKET to choose between PostgreSQL (PgAssetByteStore) and S3 (S3AssetByteStore) implementations in lib/persistence/asset-byte-store.ts.
• Lazy initialization: The lazyAssetByteStore factory defers backend instantiation until first use, preventing startup crashes from invalid S3 configuration.
• Transaction semantics: PostgreSQL storage supports transaction-pinned methods (writeWith, readWith), while S3 storage sets writesOutsideRegistryDatabase: true indicating non-transactional behavior.
• Client delivery modes: Configure ASSET_BYTE_EGRESS=redirect for signed S3 URLs or omit for direct streaming through the application server.

Frequently Asked Questions

How does OpenMAIC validate S3 bucket names?

The configuredS3Bucket function in lib/persistence/asset-byte-store.ts enforces AWS naming rules including length constraints (3–63 characters), allowed characters (lowercase letters, numbers, hyphens, periods), and prohibitions against reserved prefixes like xn-- or suffixes like -s3alias. Invalid configurations throw immediate errors with descriptive messages.

What happens if I configure an invalid S3 bucket but don't use asset features?

Due to the lazy initialization pattern in lazyAssetByteStore, the application starts successfully even with malformed ASSET_S3_BUCKET values. The error only surfaces when code first attempts to read or write an asset, isolating the failure to asset-specific requests while keeping the rest of the system operational.

Can I migrate assets from PostgreSQL to S3 after deployment?

The source code analysis does not reveal built-in migration utilities. Since PgAssetByteStore and S3AssetByteStore implement the same interface but store bytes differently, migration would require custom scripts to read from the PostgreSQL store via readWith methods and write to S3 via the S3 store, followed by updating the ASSET_S3_BUCKET environment variable.

Does OpenMAIC support storage backends other than PostgreSQL and S3?

According to the current implementation in lib/persistence/asset-byte-store.ts, only two backends are implemented: PgAssetByteStore for PostgreSQL and S3AssetByteStore for Amazon S3. The architecture uses a pluggable interface (AssetByteStore), but no additional adapters (such as Azure Blob or GCS) are present in the analyzed codebase.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}