OpenMAIC Production Asset Management Settings: Configuring Asset Collection Intervals and Storage Backends

OpenMAIC exposes all production asset management settings through environment-driven configuration in configs/storage.ts, with ASSET_COLLECTION_INTERVAL_MS controlling how frequently the server scans cloud storage for new assets.

The THU-MAIC/OpenMAIC repository manages dynamic media pipelines that require robust, production-grade asset handling. According to the OpenMAIC source code, asset lifecycle management—including collection intervals, retention policies, and cloud storage credentials—is centralized in the storage configuration layer rather than scattered across service modules.

Core Production Asset Settings in configs/storage.ts

The configs/storage.ts file serves as the single source of truth for production asset behavior. All settings are read from environment variables at startup, enabling secure, deployment-specific configuration without code changes.

Setting	Purpose	Default	Environment Variable
Collection interval	Frequency of bucket scans to discover new/modified assets	300_000 ms (5 minutes)	ASSET_COLLECTION_INTERVAL_MS
Retention window	Days before unreferenced assets are marked for garbage collection	30 days	ASSET_RETENTION_DAYS
Storage bucket	Target cloud bucket for all asset operations	none (required)	ASSET_BUCKET
Cloud region	AWS region or equivalent for bucket operations	none (required)	ASSET_REGION
Access credentials	IAM keys for authenticated storage access	none (required)	ASSET_ACCESS_KEY, ASSET_SECRET_KEY

These values flow into the /lib/asset/ subsystem, where lib/asset/collector.ts instantiates the periodic scanner and lib/asset/registry.ts maintains the in-memory asset catalog.

How the Asset Collection Interval Works

The ASSET_COLLECTION_INTERVAL_MS setting directly controls a setInterval timer created during server initialization. The implementation follows this sequence:

1. Configuration load – configs/storage.ts validates and exports the parsed interval value
2. Collector initialization – lib/asset/collector.ts receives the interval and starts the recurring task
3. Bucket enumeration – Each tick lists objects in ASSET_BUCKET, comparing metadata against the registry
4. Delta processing – New assets are ingested; modified assets trigger downstream invalidation; stale entries are flagged per ASSET_RETENTION_DAYS

The collector is non-blocking: scans run independently of request handling, with registry updates published via internal event bus for reactive consumption.

Configuring Asset Collection in Production

Operators tune the collection interval based on deployment characteristics:

• High-churn environments (frequent user uploads): Reduce to 60_000 ms (1 minute) for near-real-time asset availability
• Stable production systems: Retain the 300_000 ms default to minimize S3 LIST API costs
• Batch-import scenarios: Disable automatic collection (0 or negative value) and trigger manual scans via admin endpoint

Example production environment configuration:

# .env.production

ASSET_COLLECTION_INTERVAL_MS=60000
ASSET_RETENTION_DAYS=7
ASSET_BUCKET=openmaic-assets-prod
ASSET_REGION=ap-northeast-1
ASSET_ACCESS_KEY=AKIA...
ASSET_SECRET_KEY=...

Code Implementation Examples

Initializing the Asset Collector with Custom Interval

// configs/storage.ts — configuration definition
export const storageConfig = {
  collectionIntervalMs: Number(
    process.env.ASSET_COLLECTION_INTERVAL_MS ?? 300_000
  ),
  retentionDays: Number(process.env.ASSET_RETENTION_DAYS ?? 30),
  bucket: process.env.ASSET_BUCKET!,
  region: process.env.ASSET_REGION!,
  credentials: {
    accessKeyId: process.env.ASSET_ACCESS_KEY!,
    secretAccessKey: process.env.ASSET_SECRET_KEY!,
  },
} as const;

// lib/asset/collector.ts — timer instantiation
import { storageConfig } from '@/configs/storage';
import { scanAssetBucket } from './scanner';

export function scheduleAssetCollection(): void {
  const { collectionIntervalMs } = storageConfig;
  
  if (collectionIntervalMs <= 0) {
    console.log('[asset] Automatic collection disabled');
    return;
  }

  setInterval(async () => {
    try {
      const delta = await scanAssetBucket();
      await updateRegistry(delta);
    } catch (err) {
      console.error('[asset] Collection scan failed:', err);
    }
  }, collectionIntervalMs);
  
  console.log(`[asset] Collection scheduled every ${collectionIntervalMs}ms`);
}

Registry Lookup for Asset Consumers

// lib/asset/registry.ts — reactive asset metadata store
import { EventEmitter } from 'events';

interface AssetMeta {
  key: string;
  size: number;
  lastModified: Date;
  etag: string;
}

class AssetRegistry extends EventEmitter {
  private assets = new Map<string, AssetMeta>();

  upsert(id: string, meta: AssetMeta): void {
    this.assets.set(id, meta);
    this.emit('changed', id, meta);
  }

  get(id: string): AssetMeta | undefined {
    return this.assets.get(id);
  }

  isStale(id: string, retentionDays: number): boolean {
    const meta = this.assets.get(id);
    if (!meta) return true;
    const cutoff = Date.now() - retentionDays * 24 * 3600 * 1000;
    return meta.lastModified.getTime() < cutoff;
  }
}

export const assetRegistry = new AssetRegistry();

Storage Client Construction from Environment

// lib/asset/s3-client.ts — credential injection
import { S3 } from 'aws-sdk';
import { storageConfig } from '@/configs/storage';

export function getS3Client(): S3 {
  return new S3({
    region: storageConfig.region,
    credentials: storageConfig.credentials,
    // Production: enable signature versioning for bucket policies
    signatureVersion: 'v4',
  });
}

export function getAssetUrl(key: string): string {
  return `https://${storageConfig.bucket}.s3.${storageConfig.region}.amazonaws.com/${key}`;
}

Source File Reference Map

File Path	Responsibility
configs/storage.ts	Environment parsing and config validation
lib/asset/collector.ts	Interval-driven bucket scanning
lib/asset/scanner.ts	S3 LIST operations and pagination
lib/asset/registry.ts	In-memory asset metadata with event emission
lib/asset/s3-client.ts	Authenticated S3 client factory
lib/video-export/pipeline.ts	Consumer: resolves asset URLs at render time
lib/interactive/widgets/loader.ts	Consumer: fetches latest HTML assets on request

Summary

• Production asset settings in OpenMAIC are environment-driven, defined in configs/storage.ts
• ASSET_COLLECTION_INTERVAL_MS (default 300,000 ms) controls scan frequency for new assets
• ASSET_RETENTION_DAYS (default 30) determines garbage collection eligibility
• All cloud credentials are injected via environment variables, never hard-coded
• The /lib/asset/ subsystem provides modular collector, registry, and client implementations
• Consumers subscribe to registry events rather than polling, ensuring efficient resource usage

Frequently Asked Questions

What is the default asset collection interval in OpenMAIC?

The default interval is 300,000 milliseconds (5 minutes), specified via ASSET_COLLECTION_INTERVAL_MS. This balances timeliness against S3 API costs for typical production workloads.

How do I disable automatic asset collection entirely?

Set ASSET_COLLECTION_INTERVAL_MS=0 or any negative value. The collector initialization in lib/asset/collector.ts explicitly checks for non-positive values and skips timer creation, logging that automatic collection is disabled.

Which components consume the asset registry?

The video export pipeline (lib/video-export/pipeline.ts) and interactive widget loader (lib/interactive/widgets/loader.ts) are primary consumers. Both listen to registry change events to invalidate caches and fetch fresh asset URLs when underlying storage objects change.

Where should production credentials be configured?

Never commit credentials to source control. The configs/storage.ts module expects ASSET_ACCESS_KEY and ASSET_SECRET_KEY as environment variables. In containerized deployments, use secrets management (Kubernetes secrets, AWS Secrets Manager, or Docker secrets) to inject these values at runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}