# How Celery is Configured with Redis as a Broker in Dify

> Learn how Dify configures Celery with Redis as a broker. Discover how environment variables like CELERY_BROKER_URL and SSL options are used for efficient task management.

- Repository: [LangGenius/dify](https://github.com/langgenius/dify)
- Tags: how-to-guide
- Published: 2026-02-25

---

**Dify configures Celery with Redis as a broker through the `init_app` factory in [`api/extensions/ext_celery.py`](https://github.com/langgenius/dify/blob/main/api/extensions/ext_celery.py), which reads environment variables like `CELERY_BROKER_URL` and optionally enables SSL via `_get_celery_ssl_options()` when using `rediss://` URLs.**

The `langgenius/dify` repository uses Celery to handle asynchronous tasks such as workflow execution and document indexing. Understanding how Celery is configured with Redis as a broker in Dify requires examining the configuration models, the extension factory, and the SSL helper functions that secure the connection.

## Configuration Layer: Environment Variables and MiddlewareConfig

Dify centralizes Celery configuration inside the `MiddlewareConfig` class located in [`api/configs/middleware/__init__.py`](https://github.com/langgenius/dify/blob/main/api/configs/middleware/__init__.py). This configuration model exposes the following critical environment variables:

- `CELERY_BROKER_URL` – The Redis connection string (e.g., `redis://redis:6379/0` or `rediss://` for TLS).
- `CELERY_BACKEND` – The result backend type, often set to `redis`.
- `CELERY_USE_SENTINEL` – Boolean flag to enable Redis Sentinel support.
- `CELERY_SENTINEL_MASTER_NAME`, `CELERY_SENTINEL_SOCKET_TIMEOUT`, `CELERY_SENTINEL_PASSWORD` – Sentinel-specific parameters.

The configuration also computes the `BROKER_USE_SSL` property, which returns `True` when the broker URL starts with `rediss://`. This flag drives conditional SSL initialization later in the stack.

## SSL and Security: The _get_celery_ssl_options Helper

When running Celery against a Redis broker that requires TLS, Dify applies SSL parameters through the `_get_celery_ssl_options()` function in [`api/extensions/ext_celery.py`](https://github.com/langgenius/dify/blob/main/api/extensions/ext_celery.py).

This helper performs two validation checks:

1. Verifies `dify_config.BROKER_USE_SSL` is enabled.
2. Confirms the broker URL uses the `redis://` or `rediss://` scheme.

If both conditions pass, the function maps the string value of `REDIS_SSL_CERT_REQS` (e.g., `CERT_NONE`, `CERT_REQUIRED`) to the corresponding Python `ssl` constant. It then returns a dictionary containing:

- `ssl_cert_reqs`
- `ssl_ca_certs`
- `ssl_certfile`
- `ssl_keyfile`

These values are pulled directly from environment variables defined in the configuration layer.

## Celery App Factory: init_app and Redis Broker Setup

The core integration happens inside `init_app(app)` in [`api/extensions/ext_celery.py`](https://github.com/langgenius/dify/blob/main/api/extensions/ext_celery.py). This factory function constructs the Celery instance and binds it to the Flask application.

The factory creates the Celery object with these parameters:

```python
celery_app = Celery(
    app.name,
    task_cls=FlaskTask,  # Custom class that preserves Flask request context

    broker=dify_config.CELERY_BROKER_URL,
    backend=dify_config.CELERY_BACKEND,
)

```

After instantiation, the factory updates the configuration with:

- `result_backend` mapped from `CELERY_RESULT_BACKEND`
- `broker_transport_options` (populated conditionally for Sentinel)
- `task_ignore_result=True`
- `timezone` derived from `LOG_TZ`

### Sentinel Support for High Availability

When `CELERY_USE_SENTINEL` is enabled, Dify constructs a `broker_transport_options` dictionary containing:

- `master_name`: The Sentinel master name.
- `sentinel_kwargs`: A dictionary with `socket_timeout` and `password`.

These options are passed to `celery_app.conf.update()`, allowing Celery to discover the current Redis master through the Sentinel topology rather than a static hostname.

### SSL Injection for Encrypted Connections

After basic configuration, the factory calls `_get_celery_ssl_options()`. If SSL options are returned, the factory injects them into the Celery configuration:

```python
ssl_options = _get_celery_ssl_options()
if ssl_options:
    celery_app.conf.update(
        broker_use_ssl=ssl_options,
        redis_backend_use_ssl=(
            ssl_options if dify_config.CELERY_BACKEND == "redis" else None
        ),
    )

```

This ensures that both the broker connection and the result backend use TLS when configured with `rediss://` URLs.

Finally, the factory populates the beat schedule, imports task modules, and stores the instance in `app.extensions["celery"]` for global access.

## Runtime Task Execution

With the Celery instance configured, background work is defined in the `api/tasks/` directory. Files such as [`api/tasks/workflow_execution_tasks.py`](https://github.com/langgenius/dify/blob/main/api/tasks/workflow_execution_tasks.py) use the `@shared_task` decorator to register functions with the Celery app.

Tasks are dispatched from API endpoints or schedulers using:

```python
current_app.extensions["celery"].send_task(
    "tasks.generate_summary_index_task",
    args=[app_id],
)

```

Because the broker is configured with a Redis URL, Celery serializes the task message and pushes it to the appropriate Redis list or stream. Worker processes then fetch and execute the task, with results optionally stored back to the Redis backend.

## Monitoring and Queue Inspection

Dify includes a queue monitoring utility in [`api/schedule/queue_monitor_task.py`](https://github.com/langgenius/dify/blob/main/api/schedule/queue_monitor_task.py). This task directly instantiates a Redis client using the same `BROKER_USE_SSL` flag to inspect queue lengths outside of the Celery abstraction layer.

By reusing the SSL configuration logic, the monitor maintains consistency with the broker connection parameters, ensuring it can connect to TLS-enabled Redis instances when the main Celery broker uses `rediss://`.

## Summary

- **Configuration**: Dify reads Redis broker settings from `MiddlewareConfig` in [`api/configs/middleware/__init__.py`](https://github.com/langgenius/dify/blob/main/api/configs/middleware/__init__.py), exposing `CELERY_BROKER_URL` and `BROKER_USE_SSL`.
- **SSL Handling**: The `_get_celery_ssl_options()` function in [`api/extensions/ext_celery.py`](https://github.com/langgenius/dify/blob/main/api/extensions/ext_celery.py) conditionally builds SSL parameters when `rediss://` URLs are detected.
- **Factory Pattern**: The `init_app()` function constructs the Celery instance with Redis as the broker, injects Sentinel transport options for high availability, and applies SSL configuration to both broker and backend.
- **Task Execution**: Background jobs are defined in `api/tasks/` and dispatched via the Celery app stored in `app.extensions["celery"]`.
- **Monitoring**: Queue monitoring reuses the same SSL flags to connect directly to Redis for inspection.

## Frequently Asked Questions

### What environment variables configure the Redis broker in Dify?

Dify uses `CELERY_BROKER_URL` to define the Redis connection string (e.g., `redis://localhost:6379/0`). Optional variables include `CELERY_BACKEND` for the result store, `CELERY_USE_SENTINEL` to enable Sentinel mode, and `REDIS_SSL_CERT_REQS` along with related certificate paths when using TLS (`rediss://`).

### How does Dify handle SSL/TLS for the Celery Redis broker?

When the broker URL starts with `rediss://`, the `BROKER_USE_SSL` flag becomes true. The `_get_celery_ssl_options()` function in [`api/extensions/ext_celery.py`](https://github.com/langgenius/dify/blob/main/api/extensions/ext_celery.py) then constructs a dictionary of SSL parameters—including `ssl_cert_reqs`, `ssl_ca_certs`, and key files—which is injected into the Celery configuration as `broker_use_ssl` and `redis_backend_use_ssl`.

### Does Dify support Redis Sentinel for Celery high availability?

Yes. Setting `CELERY_USE_SENTINEL=true` triggers the `init_app` factory to populate `broker_transport_options` with the Sentinel master name and connection kwargs (socket timeout and password). This allows Celery to discover the current Redis master dynamically through the Sentinel topology rather than connecting to a static Redis host.

### Where are Celery tasks defined in the Dify codebase?

Task definitions reside in the `api/tasks/` directory. For example, [`api/tasks/workflow_execution_tasks.py`](https://github.com/langgenius/dify/blob/main/api/tasks/workflow_execution_tasks.py) contains background jobs that execute workflows or index documents. These functions use the `@shared_task` decorator and are dispatched via `current_app.extensions["celery"].send_task()` from API endpoints or the Celery beat scheduler.