Sidekiq Configuration and Background Job System in Maybe Finance

The Maybe Finance application configures Sidekiq as its background job processor through Rails ActiveJob, utilizing weighted priority queues, SHA-256 secured Web UI access, and a centralized ApplicationJob base class that defines default retry and discard policies.

The open-source personal finance platform maybe-finance/maybe relies on Sidekiq for all asynchronous processing, from account synchronization to AI chat responses. This architecture leverages ActiveJob's abstraction layer while exposing Sidekiq-specific features like priority-weighted queues and cron scheduling. Understanding this Sidekiq configuration is essential for developers extending the application's background task capabilities.

ApplicationJob Base Class Configuration

All background jobs inherit from ApplicationJob, defined in app/jobs/application_job.rb. This base class establishes universal behavior for retries, discards, and queue assignment.

# app/jobs/application_job.rb

class ApplicationJob < ActiveJob::Base
  retry_on ActiveRecord::Deadlocked
  discard_on ActiveJob::DeserializationError
  queue_as :low_priority
end

The retry_on declaration automatically re-executes jobs encountering database deadlock errors, while discard_on silently removes corrupted jobs failing deserialization. By defaulting queue_as to :low_priority, the system ensures background tasks do not block critical user-facing operations unless explicitly reassigned.

Priority Queue Configuration in sidekiq.yml

Sidekiq reads its queue structure from config/sidekiq.yml, which defines five weighted priority levels. The weight values determine the relative frequency with which Sidekiq polls each queue for available work.

# config/sidekiq.yml

concurrency: <%= ENV.fetch("RAILS_MAX_THREADS") { 3 } %>
queues:
  - [scheduled, 10]
  - [high_priority, 4]
  - [medium_priority, 2]
  - [low_priority, 1]
  - [default, 1]

The concurrency setting defaults to 3 threads but respects the RAILS_MAX_THREADS environment variable. The scheduled queue receives the highest weight (10), ensuring cron-like jobs execute promptly, while high_priority (4) handles urgent operations like financial account synchronization. Lower weights on :low_priority and :default prevent background reports or imports from overwhelming worker resources.

Cron Scheduling and Grace Period Configuration

Maybe uses the sidekiq-cron gem for periodic task execution. In config/initializers/sidekiq.rb (lines 13-16), the initializer configures a 10-minute grace period to prevent missed executions during deployments.

# config/initializers/sidekiq.rb

Sidekiq::Cron.configure do |config|
  config.reschedule_grace_period = 600
end

This reschedule_grace_period of 600 seconds ensures cron jobs scheduled to run during a server restart are rescheduled rather than skipped. Additional cron jobs register through Sidekiq::Cron::Job.create, specifying their target class and queue assignment.

Securing the Sidekiq Web Interface

In production environments, the Sidekiq Web UI requires HTTP Basic Authentication using SHA-256 hashed credentials. In config/initializers/sidekiq.rb (lines 3-10), the production environment activates this protection:

if Rails.env.production?
  Sidekiq::Web.use(Rack::Auth::Basic) do |username, password|
    configured_username = ::Digest::SHA256.hexdigest(ENV.fetch("SIDEKIQ_WEB_USERNAME", "maybe"))
    configured_password = ::Digest::SHA256.hexdigest(ENV.fetch("SIDEKIQ_WEB_PASSWORD", "maybe"))

    ActiveSupport::SecurityUtils.secure_compare(
      ::Digest::SHA256.hexdigest(username), configured_username
    ) && ActiveSupport::SecurityUtils.secure_compare(
      ::Digest::SHA256.hexdigest(password), configured_password
    )
  end
end

The system compares SHA-256 digests of provided credentials against SIDEKIQ_WEB_USERNAME and SIDEKIQ_WEB_PASSWORD environment variables using ActiveSupport::SecurityUtils.secure_compare to prevent timing attacks.

Routing and Job Execution Patterns

The Web UI mounts at /sidekiq via config/routes.rb (lines 15-17), making the dashboard accessible for monitoring queue depth and retrying failed jobs:

mount Sidekiq::Web => "/sidekiq"

Throughout the codebase, domain objects enqueue jobs via perform_later. For example, the account.sync! method enqueues a SyncJob to fetch fresh financial data, while import controllers call ImportJob.perform_later(import) to process CSV files asynchronously.

The SyncJob implementation demonstrates high-priority queue assignment for account synchronization:

# app/jobs/sync_job.rb

class SyncJob < ApplicationJob
  queue_as :high_priority

  def perform(sync)
    sync.perform
  end
end

Enqueue this job from controllers or services using:

sync = Sync.new(account)
SyncJob.perform_later(sync)

Other domain-specific jobs like AssistantResponseJob (for LLM chat generation) follow identical patterns, overriding queue_as when specific priority requirements exist.

Summary

• ApplicationJob centralizes retry logic for deadlocks and discards deserialization errors, defaulting all jobs to the :low_priority queue.
• sidekiq.yml configures five weighted queues ranging from scheduled (weight 10) to default (weight 1), with concurrency tied to RAILS_MAX_THREADS.
• sidekiq.rb initializes cron scheduling with a 600-second grace period and protects the Web UI via SHA-256-based HTTP Basic Auth in production.
• routes.rb mounts the monitoring dashboard at /sidekiq (lines 15-17) for operational visibility.
• Job execution uses perform_later on classes inheriting from ApplicationJob, with SyncJob serving as the primary example of high-priority queue assignment.

Frequently Asked Questions

How do I create a new background job in Maybe Finance?

Create a Ruby class in app/jobs/ that inherits from ApplicationJob, declare the appropriate queue_as symbol, and implement a perform method. For example, a medium-priority report generator would set queue_as :medium_priority and accept parameters like user_id in its perform method. Enqueue the job anywhere in the application by calling YourJobName.perform_later(arguments).

What is the default queue for background jobs in Maybe?

By default, all jobs queue on :low_priority as defined in the ApplicationJob base class. Individual jobs override this default by declaring queue_as :high_priority or queue_as :medium_priority within their class definition, mapping to the weighted queues defined in config/sidekiq.yml.

How does Maybe secure the Sidekiq Web UI in production?

The production environment protects the /sidekiq endpoint with HTTP Basic Authentication. The system hashes provided credentials using SHA-256 and compares them against the SIDEKIQ_WEB_USERNAME and SIDEKIQ_WEB_PASSWORD environment variables using ActiveSupport::SecurityUtils.secure_compare to prevent timing attacks.

How are cron jobs configured to handle missed executions during deploys?

The Sidekiq initializer in config/initializers/sidekiq.rb configures reschedule_grace_period = 600 (10 minutes) within the Sidekiq::Cron.configure block. This setting ensures that cron jobs scheduled to run during server restarts or deployments are automatically rescheduled for execution rather than being permanently skipped.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}