How Celery is Configured with Redis as a Broker in Dify

Question

Learn how Dify configures Celery with Redis as a broker. Discover how environment variables like CELERY_BROKER_URL and SSL options are used for efficient task management.

Accepted Answer

Dify configures Celery with Redis as a broker through the init_app factory in api/extensions/ext_celery.py, which reads environment variables like CELERY_BROKER_URL and optionally enables SSL via _get_celery_ssl_options() when using rediss:// URLs.

The langgenius/dify repository uses Celery to handle asynchronous tasks such as workflow execution and document indexing. Understanding how Celery is configured with Redis as a broker in Dify requires examining the configuration models, the extension factory, and the SSL helper functions that secure the connection.

Configuration Layer: Environment Variables and MiddlewareConfig

Dify centralizes Celery configuration inside the MiddlewareConfig class located in api/configs/middleware/__init__.py. This configuration model exposes the following critical environment variables:

CELERY_BROKER_URL – The Redis connection string (e.g., redis://redis:6379/0 or rediss:// for TLS).
CELERY_BACKEND – The result backend type, often set to redis.
CELERY_USE_SENTINEL – Boolean flag to enable Redis Sentinel support.
CELERY_SENTINEL_MASTER_NAME, CELERY_SENTINEL_SOCKET_TIMEOUT, CELERY_SENTINEL_PASSWORD – Sentinel-specific parameters.

The configuration also computes the BROKER_USE_SSL property, which returns True when the broker URL starts with rediss://. This flag drives conditional SSL initialization later in the stack.

SSL and Security: The _get_celery_ssl_options Helper

When running Celery against a Redis broker that requires TLS, Dify applies SSL parameters through the _get_celery_ssl_options() function in api/extensions/ext_celery.py.

This helper performs two validation checks:

Verifies dify_config.BROKER_USE_SSL is enabled.
Confirms the broker URL uses the redis:// or rediss:// scheme.

If both conditions pass, the function maps the string value of REDIS_SSL_CERT_REQS (e.g., CERT_NONE, CERT_REQUIRED) to the corresponding Python ssl constant. It then returns a dictionary containing:

ssl_cert_reqs
ssl_ca_certs
ssl_certfile
ssl_keyfile

These values are pulled directly from environment variables defined in the configuration layer.

Celery App Factory: init_app and Redis Broker Setup

The core integration happens inside init_app(app) in api/extensions/ext_celery.py. This factory function constructs the Celery instance and binds it to the Flask application.

The factory creates the Celery object with these parameters:

celery_app = Celery(
    app.name,
    task_cls=FlaskTask,  # Custom class that preserves Flask request context

    broker=dify_config.CELERY_BROKER_URL,
    backend=dify_config.CELERY_BACKEND,
)

After instantiation, the factory updates the configuration with:

result_backend mapped from CELERY_RESULT_BACKEND
broker_transport_options (populated conditionally for Sentinel)
task_ignore_result=True
timezone derived from LOG_TZ

Sentinel Support for High Availability

When CELERY_USE_SENTINEL is enabled, Dify constructs a broker_transport_options dictionary containing:

master_name: The Sentinel master name.
sentinel_kwargs: A dictionary with socket_timeout and password.

These options are passed to celery_app.conf.update(), allowing Celery to discover the current Redis master through the Sentinel topology rather than a static hostname.

SSL Injection for Encrypted Connections

After basic configuration, the factory calls _get_celery_ssl_options(). If SSL options are returned, the factory injects them into the Celery configuration:

ssl_options = _get_celery_ssl_options()
if ssl_options:
    celery_app.conf.update(
        broker_use_ssl=ssl_options,
        redis_backend_use_ssl=(
            ssl_options if dify_config.CELERY_BACKEND == "redis" else None
        ),
    )

This ensures that both the broker connection and the result backend use TLS when configured with rediss:// URLs.

Finally, the factory populates the beat schedule, imports task modules, and stores the instance in app.extensions["celery"] for global access.

Runtime Task Execution

With the Celery instance configured, background work is defined in the api/tasks/ directory. Files such as api/tasks/workflow_execution_tasks.py use the @shared_task decorator to register functions with the Celery app.

Tasks are dispatched from API endpoints or schedulers using:

current_app.extensions["celery"].send_task(
    "tasks.generate_summary_index_task",
    args=[app_id],
)

Because the broker is configured with a Redis URL, Celery serializes the task message and pushes it to the appropriate Redis list or stream. Worker processes then fetch and execute the task, with results optionally stored back to the Redis backend.

Monitoring and Queue Inspection

Dify includes a queue monitoring utility in api/schedule/queue_monitor_task.py. This task directly instantiates a Redis client using the same BROKER_USE_SSL flag to inspect queue lengths outside of the Celery abstraction layer.

By reusing the SSL configuration logic, the monitor maintains consistency with the broker connection parameters, ensuring it can connect to TLS-enabled Redis instances when the main Celery broker uses rediss://.

Summary

Configuration: Dify reads Redis broker settings from MiddlewareConfig in api/configs/middleware/__init__.py, exposing CELERY_BROKER_URL and BROKER_USE_SSL.
SSL Handling: The _get_celery_ssl_options() function in api/extensions/ext_celery.py conditionally builds SSL parameters when rediss:// URLs are detected.
Factory Pattern: The init_app() function constructs the Celery instance with Redis as the broker, injects Sentinel transport options for high availability, and applies SSL configuration to both broker and backend.
Task Execution: Background jobs are defined in api/tasks/ and dispatched via the Celery app stored in app.extensions["celery"].
Monitoring: Queue monitoring reuses the same SSL flags to connect directly to Redis for inspection.

Frequently Asked Questions

What environment variables configure the Redis broker in Dify?

Dify uses CELERY_BROKER_URL to define the Redis connection string (e.g., redis://localhost:6379/0). Optional variables include CELERY_BACKEND for the result store, CELERY_USE_SENTINEL to enable Sentinel mode, and REDIS_SSL_CERT_REQS along with related certificate paths when using TLS (rediss://).

How does Dify handle SSL/TLS for the Celery Redis broker?

When the broker URL starts with rediss://, the BROKER_USE_SSL flag becomes true. The _get_celery_ssl_options() function in api/extensions/ext_celery.py then constructs a dictionary of SSL parameters—including ssl_cert_reqs, ssl_ca_certs, and key files—which is injected into the Celery configuration as broker_use_ssl and redis_backend_use_ssl.

Does Dify support Redis Sentinel for Celery high availability?

Yes. Setting CELERY_USE_SENTINEL=true triggers the init_app factory to populate broker_transport_options with the Sentinel master name and connection kwargs (socket timeout and password). This allows Celery to discover the current Redis master dynamically through the Sentinel topology rather than connecting to a static Redis host.

Where are Celery tasks defined in the Dify codebase?

Task definitions reside in the api/tasks/ directory. For example, api/tasks/workflow_execution_tasks.py contains background jobs that execute workflows or index documents. These functions use the @shared_task decorator and are dispatched via current_app.extensions["celery"].send_task() from API endpoints or the Celery beat scheduler.