What Is the Role of Celery and Redis in Securo? Architecture and Implementation Guide

Celery and Redis serve as Securo's distributed task queue system, with Celery executing background jobs and Redis acting as both the message broker and result backend to enable asynchronous, scalable processing of time‑consuming operations.

Securo, an open‑source financial platform, relies on Celery and Redis to keep its API responsive while handling heavy‑duty work behind the scenes. Whether syncing bank connections, refreshing foreign‑exchange rates, or applying asset growth rules, these operations run outside the request/response cycle thanks to this asynchronous architecture. This guide examines how these components are wired together in the securo-finance/securo codebase, with concrete examples from the source files.

How Celery and Redis Fit Into Securo's Architecture

Component Responsibilities

Component	Purpose in Securo	Key Source File
Celery	Executes background tasks, provides periodic scheduling via Celery Beat, and offers a simple enqueue API (celery_app.send_task)	backend/app/worker.py
Redis	Acts as message broker (task transport) and result backend (outcome storage); centralizes state for worker coordination	backend/app/core/config.py (lines 107‑109)
Docker/Kubernetes	Orchestrates celery‑worker and celery‑beat containers as separate services	docker‑compose.yml (lines 149‑156)

When an API endpoint receives a request requiring intensive processing—such as uploading a knowledge document or syncing a banking connection—it dispatches a Celery task rather than blocking the response. Celery serializes the request as JSON, pushes it to Redis, and a worker process dequeues and executes it. Results are written back to Redis for callers to query.

Celery Configuration in Securo

The central Celery application is defined in backend/app/worker.py, where the broker and backend are configured to use the same Redis instance.

# backend/app/worker.py (simplified structure)

from celery import Celery
from app.core.config import settings

celery_app = Celery(
    "securo",
    broker=settings.redis_url,      # Redis as message broker

    backend=settings.redis_url,     # Redis as result backend

)

celery_app.conf.beat_schedule = {
    "sync-all-connections-hourly": {
        "task": "app.tasks.sync_tasks.sync_all_connections",
        "schedule": 60 * 60,  # every hour

    },
    # ... additional scheduled tasks

}

The redis_url is sourced from backend/app/core/config.py at lines 107‑109, ensuring consistent configuration across broker, backend, and any direct Redis operations.

Redis as Broker and Backend

Redis fulfills dual roles in Securo's Celery stack:

• Message broker: Transports task messages from producers (API endpoints, other services) to available workers. This decouples task submission from execution.
• Result backend: Persists task return values, exceptions, and metadata. Callers can query task status and retrieve outcomes asynchronously.

Using Redis for both purposes simplifies deployment—only one service to manage—while providing sufficient throughput and persistence for Securo's workload patterns.

Defining and Enqueuing Tasks

Enqueuing from API Endpoints

API endpoints trigger asynchronous work without waiting for completion:

# backend/app/main.py (lines 89-91, simplified)

from app.worker import celery_app

def trigger_sync():
    # Returns immediately; task executes in background

    celery_app.send_task("app.tasks.sync_tasks.sync_all_connections")

The send_task method accepts the task's fully qualified name, serializes arguments, and places the message on the Redis queue.

Task Implementation

Actual task logic resides in dedicated modules under backend/app/tasks/:

# backend/app/tasks/sync_tasks.py (lines 82-84)

from app.worker import celery_app

@celery_app.task(name="app.tasks.sync_tasks.sync_all_connections")
def sync_all_connections():
    # Heavy-weight logic contacting external banking APIs

    ...

The explicit name parameter ensures stable task identifiers across refactors, critical for reliable enqueuing and Celery Beat scheduling.

Periodic Task Scheduling with Celery Beat

Celery Beat, a built‑in scheduler, uses the same Redis store to coordinate recurring jobs across multiple replicas:

# backend/app/worker.py (lines 21-27)

celery_app.conf.beat_schedule = {
    "sync-all-connections-hourly": {
        "task": "app.tasks.sync_tasks.sync_all_connections",
        "schedule": 60 * 60,  # 3600 seconds = hourly

    },
    # ... other scheduled tasks (FX rates, asset growth rules, etc.)

}

A single celery‑beat container runs this scheduler; if deployed with multiple replicas, external locking (e.g., redis‑lock or database‑backed django‑celery‑beat) prevents duplicate executions.

Container Orchestration

Securo's docker‑compose.yml defines separate services for workers and the scheduler, each connecting to the shared Redis instance:

# docker-compose.yml (lines 149-156)

celery-worker:
  image: securo:latest
  command: celery -A app.worker worker --loglevel=info --concurrency=2
  depends_on:
    - redis

celery-beat:
  image: securo:latest
  command: celery -A app.worker beat --loglevel=info
  depends_on:
    - redis

The concurrency=2 parameter controls worker parallelism; this scales horizontally by adding more celery‑worker replicas, all consuming from the same Redis queue.

Architectural Benefits

• Decoupled execution: API returns immediately; heavy work proceeds asynchronously, improving perceived latency.
• Horizontal scalability: Add worker containers to increase throughput without code changes.
• Fault tolerance: Tasks persist in Redis; crashed workers don't lose pending jobs.
• Idempotent design: Scheduled tasks like apply_asset_growth_rules are written to run safely even if executed multiple times.

Key Files Reference

File	Role
backend/app/worker.py	Celery app instantiation, broker/backend config, beat schedule
backend/app/core/config.py	redis_url setting (lines 107‑109)
backend/app/tasks/*.py	Task implementations (sync, FX, asset growth, etc.)
docker-compose.yml	Service definitions for celery-worker, celery-beat, redis
charts/securo/templates/	Kubernetes equivalents (Helm charts)

Summary

• Celery provides Securo's asynchronous task execution and periodic scheduling infrastructure.
• Redis functions as both message broker and result backend, unifying state management for the task queue.
• Tasks are defined in backend/app/tasks/ and enqueued via celery_app.send_task() from API endpoints.
• docker-compose.yml orchestrates separate celery-worker and celery-beat containers for scalable, resilient background processing.

Frequently Asked Questions

What is the role of Redis in Securo's Celery setup?

Redis serves two purposes: as the message broker that routes tasks from producers to workers, and as the result backend that stores task outcomes for later retrieval. Both roles use the same redis_url configured in backend/app/core/config.py (lines 107‑109), simplifying deployment while providing the persistence and throughput Securo requires.

How does Securo schedule recurring background tasks?

Securo uses Celery Beat, configured in backend/app/worker.py (lines 21‑27) via the beat_schedule dictionary. The beat process runs as a separate container (celery-beat in docker-compose.yml) and pushes scheduled task messages to Redis at defined intervals, where workers pick them up for execution.

Can Securo scale Celery workers horizontally?

Yes. The docker-compose.yml defines stateless celery-worker containers that consume from the shared Redis queue. Increasing replica count distributes load automatically, with no code changes required. The --concurrency parameter controls threads per worker; combine with horizontal scaling for fine‑grained throughput control.

Where are Celery tasks defined in the Securo repository?

Task implementations reside in backend/app/tasks/ (e.g., sync_tasks.py). Each task uses the @celery_app.task decorator with an explicit name parameter for stable referencing. The central Celery application is defined in backend/app/worker.py, which configures the broker, backend, and periodic schedule.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}