# How to Configure Server-Backed Persistence with Postgres and S3 Artifact Storage in OpenMAIC

> Learn how to configure server-backed persistence with Postgres and S3 artifact storage in OpenMAIC. Streamline your data management and enhance scalability today.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: how-to-guide
- Published: 2026-09-13

---

**Configure the `DATABASE_URL` environment variable for PostgreSQL connectivity and set `MAIC_ARTIFACT_S3_*` variables to enable S3 offload; the runtime validates these at startup and routes binary assets to S3 when configured, otherwise storing them as PostgreSQL `BYTEA` columns.**

The THU-MAIC/OpenMAIC platform separates structured user data from binary artifacts by defaulting to PostgreSQL for relational storage while optionally streaming large objects to S3-compatible object stores. This architecture prevents database bloat while maintaining ACID compliance for metadata and session state.

## Configure PostgreSQL as the Primary Data Store

### Set the Database Connection String

OpenMAIC expects a standard PostgreSQL connection string via the `DATABASE_URL` environment variable. The configuration validator located in [`tests/server/config-validation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/server/config-validation.test.ts) (lines 200-257) parses this string during server initialization to ensure proper formatting before the runtime attempts to connect.

```bash
export DATABASE_URL="postgres://user:password@localhost:5432/openmaic"

```

### Runtime Storage Adapter

The storage layer wires the PostgreSQL client through [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts), which adapts the `node-postgres` pool to the generic storage contract used by the rest of the application. Low-level query execution and transaction scoping are handled in `packages/@openmaic/storage/src/runtime/pg.ts`, where the `TransactionScope` interface ensures ACID compliance across operations.

When the server starts, it instantiates the pool and executes a validation query. If the connection fails, the process exits immediately with a descriptive error, preventing the server from accepting requests against an unreachable database.

## Enable S3 Artifact Storage

### Required Environment Variables

Binary assets—such as uploaded images, generated media, or document attachments—can be stored outside the relational database to reduce I/O overhead. When the following environment variables are present, the system initializes an AWS SDK client and streams uploads directly to the specified bucket rather than inserting them as `BYTEA` columns:

- `MAIC_ARTIFACT_S3_ENDPOINT`: URL of the S3 endpoint (e.g., `https://s3.amazonaws.com` or `https://minio.example.com:9000`)
- `MAIC_ARTIFACT_S3_BUCKET`: Target bucket name
- `MAIC_ARTIFACT_S3_ACCESS_KEY_ID`: IAM access key
- `MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY`: IAM secret key
- `MAIC_ARTIFACT_S3_REGION`: Optional region specifier (defaults to `us-east-1`)

```bash
export MAIC_ARTIFACT_S3_ENDPOINT="https://s3.amazonaws.com"
export MAIC_ARTIFACT_S3_BUCKET="maic-artifacts"
export MAIC_ARTIFACT_S3_ACCESS_KEY_ID="AKIAIOSFODNN7EXAMPLE"
export MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY="wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

```

### Storage Routing Logic

The decision logic resides in `packages/@openmaic/storage/src/document/pg.ts`. This module checks for S3 configuration at startup. If detected, it overrides the default blob storage methods to use S3 streaming; otherwise, it falls back to PostgreSQL `BYTEA` storage.

This routing behavior is integration-tested in [`tests/persistence/asset-collector-schedule.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/persistence/asset-collector-schedule.test.ts) (around line 243), where the harness verifies that assets are correctly written to the configured backend and that the system gracefully handles credential rotations.

## Complete Configuration Example

### Docker Compose Deployment

For production deployments using Docker, define the persistence layer as follows:

```yaml
services:
  app:
    image: openmaic/server:latest
    environment:
      - DATABASE_URL=postgres://maic:secret@postgres:5432/maicdb
      - MAIC_ARTIFACT_S3_ENDPOINT=https://s3.amazonaws.com
      - MAIC_ARTIFACT_S3_BUCKET=maic-production-artifacts
      - MAIC_ARTIFACT_S3_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
      - MAIC_ARTIFACT_S3_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

```

### Verification Steps

1. Start the server and observe the logs for the message confirming successful database connection.
2. Look for the artifact store mode indicator: `Artifact store: S3 (bucket: maic-production-artifacts)`.
3. Upload a test asset via the UI and verify its presence in the S3 console or, if S3 is disabled, query the `maicdb` PostgreSQL database to confirm the `BYTEA` entry exists.

## Key Implementation Files

- **[`configs/storage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/configs/storage.ts)**: Defines key constants for local storage identifiers; persistence layer reads `DATABASE_URL` from the environment.
- **[`tests/server/config-validation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/server/config-validation.test.ts)**: Validates `DATABASE_URL` format and required environment variables at startup.
- **[`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts)**: Adapts the PostgreSQL pool to the generic storage interface used by the runtime.
- **`packages/@openmaic/storage/src/runtime/pg.ts`**: Implements low-level PostgreSQL query execution and transaction management.
- **`packages/@openmaic/storage/src/document/pg.ts`**: Routes binary storage between PostgreSQL `BYTEA` and S3 based on environment configuration.
- **[`tests/persistence/asset-collector-schedule.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/persistence/asset-collector-schedule.test.ts)**: Integration tests covering both PostgreSQL and S3 artifact persistence paths.

## Summary

- **PostgreSQL** is configured via the `DATABASE_URL` environment variable and validated at startup in [`config-validation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/config-validation.test.ts).
- **S3 artifact storage** activates when `MAIC_ARTIFACT_S3_ENDPOINT` and related credentials are provided, offloading binary data from the database to reduce bloat.
- The storage routing logic in `packages/@openmaic/storage/src/document/pg.ts` automatically selects between `BYTEA` columns and S3 streaming uploads.
- Connection pooling and transaction safety are enforced in [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts) using the `TransactionScope` interface from the runtime adapter.

## Frequently Asked Questions

### What happens if S3 environment variables are missing?

If the `MAIC_ARTIFACT_S3_*` variables are not defined, the system defaults to storing binary artifacts as `BYTEA` columns within the PostgreSQL database configured by `DATABASE_URL`. This fallback is handled transparently in `packages/@openmaic/storage/src/document/pg.ts` without requiring code changes.

### Does OpenMAIC support S3-compatible alternatives like MinIO?

Yes. Set `MAIC_ARTIFACT_S3_ENDPOINT` to your MinIO or other S3-compatible service URL (e.g., `https://minio.example.com:9000`). The underlying AWS SDK client accepts any endpoint implementing the S3 API protocol, allowing local or private object storage deployments.

### How does the server validate the database connection on startup?

The server executes a validation routine defined in [`tests/server/config-validation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/server/config-validation.test.ts) (lines 200-257) that parses the `DATABASE_URL` format and attempts a lightweight connection probe. If the database is unreachable or the URL is malformed, the process exits immediately with a descriptive error before accepting any user sessions.

### Where is transaction safety enforced for persisted data?

Transaction scoping is implemented in `packages/@openmaic/storage/src/runtime/pg.ts` through the `TransactionScope` interface. The adapter in [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts) wraps database operations in these scopes, ensuring that multi-step persistence operations—such as user session updates combined with artifact metadata storage—remain atomic and consistent.