How to Scale LMForge Horizontally with Multiple Celery Worker Nodes

Scale LMForge horizontally by running additional stateless Celery worker containers that connect to the shared Redis broker, allowing automatic distribution of asynchronous tasks across the worker pool without modifying the Flask API.

LMForge is an end-to-end LLMOps platform that processes asynchronous workloads—such as data ingestion, model fine-tuning, and inference pipelines—through Celery tasks. To handle increased load, you can scale LMForge horizontally by deploying multiple Celery worker nodes that share the same task queue and result backend. This architecture leverages the stateless nature of Celery workers and the centralized Redis broker to enable seamless scaling from a single development instance to a production-grade cluster.

Understanding LMForge's Celery Architecture

Before scaling, it is essential to understand how LMForge integrates Celery with its Flask application and where tasks are defined.

The Celery Extension

The Celery application is initialized in api/internal/extension/celery_extension.py. When the Flask app starts, this extension creates a Celery instance, loads the broker and result-backend settings from the Flask configuration, and attaches the Celery app to app.extensions["celery"]. This design allows the Flask API to enqueue tasks while workers consume them independently.

Task Registration

Tasks are registered in the api/internal/task/ package. Key task modules include:

• document_task.py – Handles document processing workflows
• dataset_task.py – Manages dataset operations and transformations
• app_task.py – Executes core application-level tasks
• demo_task.py – Provides simple demonstration tasks for testing

Each task is decorated with @shared_task, enabling automatic registration with any Celery app that imports the module.

Configuration

The Celery configuration is defined in api/config/default_config.py within the CELERY dictionary:

CELERY = {
    "broker_url": "redis://redis:6379/0",
    "result_backend": "redis://redis:6379/1",
    "task_serializer": "json",
    "result_serializer": "json",
    "accept_content": ["json"],
    "timezone": "UTC",
    "enable_utc": True,
}

This configuration ensures all workers connect to the same Redis instance for task distribution and result storage.

Deploying Multiple Celery Worker Nodes

Horizontal scaling requires running multiple worker processes that connect to the shared infrastructure. LMForge supports this through Docker Compose and manual deployment.

Docker Compose Configuration

The docker/docker-compose.yaml file defines the celery_worker service. This service uses the same API image as the Flask application but overrides the command to start a Celery worker instead of the HTTP server:

services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"

  api:
    build: ./api
    environment:
      - FLASK_ENV=production
    depends_on:
      - redis

  celery_worker:
    build: ./api
    command: >
      celery -A api.internal.extension.celery_extension.celery_app worker
      --loglevel=info
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1
    depends_on:
      - redis

Scaling Workers with Docker Compose

To scale LMForge horizontally, use Docker Compose's built-in scaling capability. The following command launches five identical Celery worker containers:

docker compose up -d --scale celery_worker=5

Each container automatically registers with the Redis broker and begins consuming tasks from the shared queue. Because workers are stateless, you can adjust the scale factor dynamically based on workload demands without restarting the Flask API or Redis services.

Manual Worker Startup

For environments not using Docker Compose, start additional workers manually by specifying the Celery application path:

celery -A api.internal.extension.celery_extension.celery_app worker \
       --loglevel=info \
       --concurrency=4 \
       -Q default

Ensure each worker has network access to the Redis instance defined in the configuration and imports the task modules from api/internal/task/ to register the shared tasks.

Architecture Benefits for Horizontal Scaling

LMForge's Celery implementation provides specific advantages for horizontal scaling:

• Stateless Workers: Workers do not maintain local state between tasks. They fetch configuration from environment variables and tasks from Redis, allowing instant replication.
• Shared Task Queue: All workers connect to the same Redis broker (redis:6379/0), ensuring automatic load balancing across the worker pool.
• Independent Scaling: The Flask API in api/app/http/app.py remains unaware of worker count. It enqueues tasks via app.extensions["celery"] regardless of how many workers are available.
• Resilient Processing: If a worker container fails, Redis retains the task until another worker acknowledges it, preventing data loss during scaling operations.

Summary

Scaling LMForge horizontally with multiple Celery worker nodes requires:

• Running additional worker containers that connect to the shared Redis broker defined in api/config/default_config.py
• Ensuring all workers import the task modules from api/internal/task/ to register shared tasks
• Using Docker Compose's --scale option or manual worker startup commands to adjust capacity dynamically
• Maintaining stateless worker architecture to enable seamless replication without API restarts

This approach allows LMForge to handle increased asynchronous workloads—from document ingestion to model fine-tuning—by simply adding more worker nodes to the cluster.

Frequently Asked Questions

How does LMForge distribute tasks across multiple Celery workers?

LMForge uses Redis as a centralized broker. When the Flask API calls demo_task.delay() or any task method, Celery serializes the task and pushes it to a Redis queue. All connected workers poll this same queue; the first available worker claims and executes the task, providing automatic load balancing without additional configuration.

What configuration changes are needed to add more workers?

No configuration changes are required within the LMForge codebase. Workers read the broker URL and result backend from environment variables or the Flask config (CELERY_BROKER_URL and CELERY_RESULT_BACKEND). As long as new workers can reach the Redis instance and import the task modules from api/internal/task/, they join the cluster automatically.

Can I scale workers independently of the Flask API?

Yes. The Flask API and Celery workers are separate processes. You can scale celery_worker containers up or down using docker compose up -d --scale celery_worker=N without restarting the api service. The Flask application remains available to accept HTTP requests while the worker pool adjusts to handle the processing load.

How do I monitor multiple Celery workers in LMForge?

You can monitor the worker pool using Celery's built-in inspection tools or by integrating Flower, a real-time Celery monitoring web application. Run celery -A api.internal.extension.celery_extension.celery_app inspect active to see currently processing tasks across all workers, or deploy Flower alongside your workers to visualize queue depths, task rates, and worker health in a dashboard.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}