How Plane Handles Background Processing and Asynchronous Tasks Using Celery

Plane offloads long-running operations to Celery workers with Redis as the broker, using Django integration for task discovery and django-celery-beat for database-backed periodic scheduling.

Plane, the open-source project management platform, maintains API responsiveness by delegating CPU-intensive work to background workers via Celery. This architecture separates time-consuming processes—such as email notifications, data exports, and cleanup routines—from the synchronous request-response cycle. The implementation centers on a centralized Celery configuration that auto-discovers tasks from modular background task directories within the Django codebase.

Celery Application Configuration

The core integration lives in apps/api/plane/celery.py, where the Celery application instance is initialized and bound to Django's settings infrastructure. This file establishes the fundamental connection between the task queue and the web framework.

# apps/api/plane/celery.py

from celery import Celery

app = Celery("plane")
app.config_from_object("django.conf:settings", namespace="CELERY")

The configuration reads all Celery-specific settings from the Django configuration object using the CELERY_ namespace prefix. This allows environment-specific broker URLs, result backends, and task routing rules to be managed through standard Django settings files. The apps/api/plane/__init__.py file exposes the celery_app variable at the package level, ensuring Django discovers the application during startup.

Redis Broker and Structured Logging

Plane connects to Redis via the redis_instance() utility imported from plane.settings.redis, providing a high-performance message broker for task queuing. The configuration also implements JSON-structured logging through pythonjsonlogger, ensuring that worker output is parseable by centralized logging systems in containerized deployments.

Task Definition and Discovery

Background logic is organized within the apps/api/plane/bgtasks/ directory, with each module containing domain-specific asynchronous operations.

Creating Tasks with @shared_task

Tasks are defined using the @shared_task decorator, which makes functions automatically discoverable without explicit registration in the Celery configuration. For example, the email notification system implements stack_email_notification in apps/api/plane/bgtasks/email_notification_task.py:

# apps/api/plane/bgtasks/email_notification_task.py

from celery import shared_task

@shared_task
def stack_email_notification():
    # Implementation for queuing email notifications

    pass

The decorator binds the function to the default Celery app, enabling asynchronous execution via .delay() or .apply_async() methods. Developers can enqueue this task from views or models without blocking the main thread:

# Enqueue the task asynchronously

result = stack_email_notification.delay()

Automatic Task Discovery

The app.autodiscover_tasks() call in celery.py traverses all installed Django apps to register decorated functions. This eliminates manual import maintenance as the codebase scales, automatically picking up new tasks added to any bgtasks module.

Periodic Task Scheduling with Celery Beat

For cron-like operations, Plane configures the beat_schedule dictionary in celery.py to define recurring intervals using crontab expressions. The system swaps the default scheduler for django-celery-beat's database-backed implementation.

# apps/api/plane/celery.py

from celery.schedules import crontab

app.conf.beat_schedule = {
    "check-every-five-minutes-to-send-email-notifications": {
        "task": "plane.bgtasks.email_notification_task.stack_email_notification",
        "schedule": crontab(minute="*/5"),
    },
}
app.conf.beat_scheduler = "django_celery_beat.schedulers.DatabaseScheduler"

Using DatabaseScheduler enables dynamic schedule modifications through the Django admin interface rather than requiring static configuration file updates and container restarts. Dependencies including celery==5.5.3, django_celery_beat, and django-celery-results are declared in apps/api/requirements/base.txt.

Deployment Architecture

Production deployments utilize separate Docker containers for different Celery processes, defined in shell scripts within the repository.

Worker Processes

The worker container executes the task consumers via apps/api/bin/docker-entrypoint-worker.sh:

celery -A plane worker -l info

This command starts worker processes that monitor Redis queues and execute tasks concurrently.

Beat Scheduler

The scheduler container runs the periodic task enqueuer via apps/api/bin/docker-entrypoint-beat.sh:

celery -A plane beat -l info

This process is responsible for pushing periodic tasks into the queue according to the beat_schedule configuration, while the DatabaseScheduler persists schedule state in the Django database.

Testing Background Tasks

The test suite provides isolation for unit tests through a mock_celery fixture defined in apps/api/plane/tests/conftest_external.py. This fixture patches celery.app.task.Task.delay to prevent actual background execution while allowing assertions on task invocation:

def test_email_notification_queued(client, mock_celery):
    response = client.post("/api/notifications/trigger/")
    assert response.status_code == 200
    mock_celery.assert_called_once()

This approach ensures tests run quickly without requiring a live Redis instance or worker processes, while still verifying that tasks are properly queued from the application code.

Summary

• Plane uses Celery integrated with Django via apps/api/plane/celery.py to handle asynchronous workloads outside the request cycle
• Tasks are defined in apps/api/plane/bgtasks/ using the @shared_task decorator and automatically discovered at startup
• Redis serves as the high-performance message broker for task queuing and result storage
• Periodic tasks are managed through django-celery-beat with a DatabaseScheduler for dynamic schedule editing via the admin interface
• Separate Docker containers defined in docker-entrypoint-worker.sh and docker-entrypoint-beat.sh run workers and the scheduler for horizontal scalability
• The mock_celery fixture in conftest_external.py enables isolated unit testing without live worker dependencies

Frequently Asked Questions

What message broker does Plane use for Celery tasks?

Plane uses Redis as the message broker, configured through the redis_instance() utility in the Django settings. This provides fast in-memory queuing suitable for high-throughput task processing, with the connection details managed through standard Django configuration using the CELERY_ namespace.

How are periodic background tasks scheduled in Plane?

Periodic tasks are defined in the beat_schedule dictionary within apps/api/plane/celery.py using crontab expressions for complex scheduling. The system uses django_celery_beat.schedulers.DatabaseScheduler instead of the default file-based scheduler, allowing teams to modify schedules dynamically through the Django admin interface without restarting containers.

Where should new background tasks be defined in the Plane codebase?

New asynchronous tasks should be created as Python functions decorated with @shared_task inside modules under apps/api/plane/bgtasks/. The autodiscover_tasks() mechanism in celery.py automatically registers these functions with the Celery app, making them available for execution via .delay() or .apply_async() without manual imports.

How does Plane handle Celery task testing during development?

The test suite provides a mock_celery fixture in apps/api/plane/tests/conftest_external.py that patches the delay method on Celery tasks. This allows unit tests to verify that tasks are properly queued from views and models while executing synchronously within the test process, eliminating dependencies on running Redis or worker containers during test execution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}