How LMForge Handles Data Persistence and Backups with Docker Volumes

LMForge mounts host directories as Docker volumes for every stateful component, ensuring data survives container restarts while enabling simple filesystem-based backups.

LMForge is an end-to-end LLMOps platform for multi-model agents that relies entirely on Docker volumes for data persistence. By mapping host-side directories into containers, the platform guarantees that PostgreSQL databases, Redis caches, Weaviate vector stores, and user-uploaded datasets remain intact even when containers are destroyed or upgraded.

Docker Volume Architecture in LMForge

The persistence strategy is defined in docker/docker-compose.yaml, where every service mounts specific host paths to container paths. This approach separates the application lifecycle from data lifecycle, allowing operators to back up, migrate, or restore individual components without touching container images.

Component Volume Mapping

LMForge allocates dedicated directories under ./volumes/ for each persistent service:

Component	Host Directory	Container Path	Stored Data
API	./volumes/app/storage	/app/api/storage	Dataset files, uploaded assets, temporary processing files
PostgreSQL	./volumes/db/data	/var/lib/postgresql/data/pgdata	Relational tables, schema migrations, transaction logs
Redis	./volumes/redis/data	/data	Cache snapshots and AOF persistence files
Weaviate	./volumes/weaviate	/var/lib/weaviate	Vector indexes and metadata stores

These mounts ensure that when you run docker compose down, the data remains in ./volumes/ on the host filesystem, ready for the next docker compose up.

Log Rotation and Retention Policies

Beyond database persistence, LMForge implements automatic log retention to prevent disk exhaustion. The API service writes logs to a dedicated storage directory that is created at runtime if missing.

Automatic Log Backup Configuration

In api/internal/extension/logging_extension.py, the application initializes a ConcurrentTimedRotatingFileHandler that rotates logs at midnight and maintains a 30-day history:

import os
import logging
from concurrent_log_handler import ConcurrentTimedRotatingFileHandler

def setup_logging():
    log_folder = os.path.join(os.getcwd(), "storage", "log")
    if not os.path.exists(log_folder):
        os.makedirs(log_folder)  # Runtime directory creation

    
    log_file = os.path.join(log_folder, "app.log")
    handler = ConcurrentTimedRotatingFileHandler(
        log_file,
        when='midnight',
        interval=1,
        backupCount=30,  # Retain 30 days of logs

        encoding='utf-8'
    )
    
    formatter = logging.Formatter(
        "[%(asctime)s.%(msecs)03d] %(filename)s -> %(funcName)s line:%(lineno)d [%(levelname)s]: %(message)s"
    )
    handler.setFormatter(formatter)
    
    logger = logging.getLogger()
    logger.setLevel(logging.DEBUG)
    logger.addHandler(handler)

The backupCount=30 parameter implements a retention policy without requiring external cron jobs or cleanup scripts. When the 31st daily log file is created, the oldest is automatically deleted.

Backup Strategies for LMForge Docker Volumes

Because LMForge stores all stateful data in host-mounted directories under ./volumes/, you can implement backups using standard filesystem tools without requiring database-specific dump commands.

Filesystem Snapshots

The simplest approach archives the entire volumes directory while containers are running (hot backup), since Docker volume mounts are atomic at the filesystem level:

#!/usr/bin/env bash
set -euo pipefail

BACKUP_ROOT="${PWD}/backup"
mkdir -p "$BACKUP_ROOT"

# Create timestamped archives of each volume

tar czf "$BACKUP_ROOT/pg_$(date +%F).tar.gz" -C ./volumes/db data
tar czf "$BACKUP_ROOT/redis_$(date +%F).tar.gz" -C ./volumes/redis data
tar czf "$BACKUP_ROOT/storage_$(date +%F).tar.gz" -C ./volumes/app storage
tar czf "$BACKUP_ROOT/weaviate_$(date +%F).tar.gz" -C ./volumes/weaviate .

Docker Native Commands

For point-in-time consistency, you can use docker cp to extract data from running containers without direct host path access:

# Copy PostgreSQL data directory from container to host backup location

docker cp llmops-db:/var/lib/postgresql/data/pgdata ./backup/pgdata-$(date +%F)

# Copy Redis persistence files

docker cp llmops-redis:/data ./backup/redis-$(date +%F)

Volume-Level Backup Tools

Production deployments often use specialized tools that handle Docker volumes natively:

• Restic or Borg: Point at ./volumes/ for deduplicated, encrypted backups
• Volume plugins: Use vieux/sshfs or local-persist driver for network-attached storage
• ZFS/Btrfs snapshots: If the host filesystem supports copy-on-write, snapshot the ./volumes dataset instantly

Key Configuration Files

Understanding these files helps you customize persistence behavior:

File	Purpose
docker/docker-compose.yaml	Declares all volume mounts for PostgreSQL, Redis, Weaviate, and API storage
api/internal/extension/logging_extension.py	Configures ConcurrentTimedRotatingFileHandler with 30-day retention
api/.dockerignore	Excludes storage/log and temporary files from the Docker image build, ensuring only runtime data persists in volumes
docker/postgres/init.sql	Runs only on fresh volumes; skipped if ./volumes/db/data already contains data, protecting existing databases during redeploys

Summary

• LMForge uses Docker volumes mapped to ./volumes/ subdirectories to persist PostgreSQL, Redis, Weaviate, and user-uploaded data across container restarts.
• Volume definitions live in docker/docker-compose.yaml, binding host paths like ./volumes/db/data to container paths like /var/lib/postgresql/data/pgdata.
• Log retention is handled automatically by ConcurrentTimedRotatingFileHandler in api/internal/extension/logging_extension.py, keeping 30 days of rotated logs without manual cleanup.
• Backups are filesystem-native—you can archive ./volumes/ using tar, docker cp, or tools like Restic, since all state exists on the host filesystem.

Frequently Asked Questions

Where does LMForge store PostgreSQL data?

LMForge stores PostgreSQL data in ./volumes/db/data on the host, which is mounted to /var/lib/postgresql/data/pgdata inside the container. This mapping is defined in docker/docker-compose.yaml and ensures that database files survive container recreation.

How long does LMForge retain application logs?

The platform retains application logs for 30 days. The ConcurrentTimedRotatingFileHandler configured in api/internal/extension/logging_extension.py rotates logs daily at midnight and automatically deletes files older than the 30-day backupCount limit.

Can I back up LMForge volumes while containers are running?

Yes. Because LMForge uses bind mounts to host directories rather than Docker's internal volume driver, you can safely create hot backups using tar, rsync, or snapshot tools while containers remain running. The filesystem handles consistency for the mounted directories.

What happens to data during a container restart?

Data persists completely. Since all stateful information resides in host-mounted volumes under ./volumes/, restarting or recreating containers with docker compose down && docker compose up does not affect the stored data. PostgreSQL, Redis, Weaviate, and uploaded files remain intact.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}