How to Configure Server-Backed Persistence with Postgres and S3 Artifact Storage in OpenMAIC

Configure the DATABASE_URL environment variable for PostgreSQL connectivity and set MAIC_ARTIFACT_S3_* variables to enable S3 offload; the runtime validates these at startup and routes binary assets to S3 when configured, otherwise storing them as PostgreSQL BYTEA columns.

The THU-MAIC/OpenMAIC platform separates structured user data from binary artifacts by defaulting to PostgreSQL for relational storage while optionally streaming large objects to S3-compatible object stores. This architecture prevents database bloat while maintaining ACID compliance for metadata and session state.

Configure PostgreSQL as the Primary Data Store

Set the Database Connection String

OpenMAIC expects a standard PostgreSQL connection string via the DATABASE_URL environment variable. The configuration validator located in tests/server/config-validation.test.ts (lines 200-257) parses this string during server initialization to ensure proper formatting before the runtime attempts to connect.

export DATABASE_URL="postgres://user:password@localhost:5432/openmaic"

Runtime Storage Adapter

The storage layer wires the PostgreSQL client through lib/server/agent-runtime/store.ts, which adapts the node-postgres pool to the generic storage contract used by the rest of the application. Low-level query execution and transaction scoping are handled in packages/@openmaic/storage/src/runtime/pg.ts, where the TransactionScope interface ensures ACID compliance across operations.

When the server starts, it instantiates the pool and executes a validation query. If the connection fails, the process exits immediately with a descriptive error, preventing the server from accepting requests against an unreachable database.

Enable S3 Artifact Storage

Required Environment Variables

Binary assets—such as uploaded images, generated media, or document attachments—can be stored outside the relational database to reduce I/O overhead. When the following environment variables are present, the system initializes an AWS SDK client and streams uploads directly to the specified bucket rather than inserting them as BYTEA columns:

• MAIC_ARTIFACT_S3_ENDPOINT: URL of the S3 endpoint (e.g., https://s3.amazonaws.com or https://minio.example.com:9000)
• MAIC_ARTIFACT_S3_BUCKET: Target bucket name
• MAIC_ARTIFACT_S3_ACCESS_KEY_ID: IAM access key
• MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY: IAM secret key
• MAIC_ARTIFACT_S3_REGION: Optional region specifier (defaults to us-east-1)

export MAIC_ARTIFACT_S3_ENDPOINT="https://s3.amazonaws.com"
export MAIC_ARTIFACT_S3_BUCKET="maic-artifacts"
export MAIC_ARTIFACT_S3_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

Storage Routing Logic

The decision logic resides in packages/@openmaic/storage/src/document/pg.ts. This module checks for S3 configuration at startup. If detected, it overrides the default blob storage methods to use S3 streaming; otherwise, it falls back to PostgreSQL BYTEA storage.

This routing behavior is integration-tested in tests/persistence/asset-collector-schedule.test.ts (around line 243), where the harness verifies that assets are correctly written to the configured backend and that the system gracefully handles credential rotations.

Complete Configuration Example

Docker Compose Deployment

For production deployments using Docker, define the persistence layer as follows:

services:
  app:
    image: openmaic/server:latest
    environment:
      - DATABASE_URL=postgres://maic:secret@postgres:5432/maicdb
      - MAIC_ARTIFACT_S3_ENDPOINT=https://s3.amazonaws.com
      - MAIC_ARTIFACT_S3_BUCKET=maic-production-artifacts
      - MAIC_ARTIFACT_S3_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

Verification Steps

1. Start the server and observe the logs for the message confirming successful database connection.
2. Look for the artifact store mode indicator: Artifact store: S3 (bucket: maic-production-artifacts).
3. Upload a test asset via the UI and verify its presence in the S3 console or, if S3 is disabled, query the maicdb PostgreSQL database to confirm the BYTEA entry exists.

Key Implementation Files

• configs/storage.ts: Defines key constants for local storage identifiers; persistence layer reads DATABASE_URL from the environment.
• tests/server/config-validation.test.ts: Validates DATABASE_URL format and required environment variables at startup.
• lib/server/agent-runtime/store.ts: Adapts the PostgreSQL pool to the generic storage interface used by the runtime.
• packages/@openmaic/storage/src/runtime/pg.ts: Implements low-level PostgreSQL query execution and transaction management.
• packages/@openmaic/storage/src/document/pg.ts: Routes binary storage between PostgreSQL BYTEA and S3 based on environment configuration.
• tests/persistence/asset-collector-schedule.test.ts: Integration tests covering both PostgreSQL and S3 artifact persistence paths.

Summary

• PostgreSQL is configured via the DATABASE_URL environment variable and validated at startup in config-validation.test.ts.
• S3 artifact storage activates when MAIC_ARTIFACT_S3_ENDPOINT and related credentials are provided, offloading binary data from the database to reduce bloat.
• The storage routing logic in packages/@openmaic/storage/src/document/pg.ts automatically selects between BYTEA columns and S3 streaming uploads.
• Connection pooling and transaction safety are enforced in lib/server/agent-runtime/store.ts using the TransactionScope interface from the runtime adapter.

Frequently Asked Questions

What happens if S3 environment variables are missing?

If the MAIC_ARTIFACT_S3_* variables are not defined, the system defaults to storing binary artifacts as BYTEA columns within the PostgreSQL database configured by DATABASE_URL. This fallback is handled transparently in packages/@openmaic/storage/src/document/pg.ts without requiring code changes.

Does OpenMAIC support S3-compatible alternatives like MinIO?

Yes. Set MAIC_ARTIFACT_S3_ENDPOINT to your MinIO or other S3-compatible service URL (e.g., https://minio.example.com:9000). The underlying AWS SDK client accepts any endpoint implementing the S3 API protocol, allowing local or private object storage deployments.

How does the server validate the database connection on startup?

The server executes a validation routine defined in tests/server/config-validation.test.ts (lines 200-257) that parses the DATABASE_URL format and attempts a lightweight connection probe. If the database is unreachable or the URL is malformed, the process exits immediately with a descriptive error before accepting any user sessions.

Where is transaction safety enforced for persisted data?

Transaction scoping is implemented in packages/@openmaic/storage/src/runtime/pg.ts through the TransactionScope interface. The adapter in lib/server/agent-runtime/store.ts wraps database operations in these scopes, ensuring that multi-step persistence operations—such as user session updates combined with artifact metadata storage—remain atomic and consistent.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}