Debugging Strategies for Immich Background Job Queue Issues: A Complete Guide

To debug Immich's background job queue system, trace the flow from JobService.create() through JobRepository.queue() to the BullMQ workers, monitoring JobRun events and handler returns in job.service.ts and job.repository.ts.

Immich processes long-running tasks—thumbnail generation, facial recognition, and metadata extraction—through a BullMQ-backed job queue orchestrated by JobService and JobRepository. Understanding the interaction between these two classes is essential for debugging why jobs fail to enqueue, execute, or trigger follow-up work.

How the Immich Job Queue Works

The queue system follows a strict pipeline from creation to completion:

Step	Component	Action	Source
Enqueue	JobService.create() → JobRepository.queue()	Client sends JobCreateDto; asJobItem() maps manual names to internal JobName values.	job.service.ts:L45-L47
Storage	JobRepository.queueAll()	Groups jobs by QueueName (from @OnJob decorator) and adds to BullMQ via add() or addBulk().	job.repository.ts:L60-L84
Workers	JobRepository.startWorkers()	Instantiates BullMQ Worker per queue; each re-emits JobRun event for JobService to handle.	job.repository.ts:L87-L96
Execution	JobService.onJobRun()	Emits JobStart, executes handler via JobRepository.run(), then emits JobSuccess or JobError. Always emits JobComplete.	job.service.ts:L49-L63
Follow-ups	JobService.onDone()	Based on original job name, queues follow-up jobs (e.g., thumbnail generation after upload).	job.service.ts:L68-L122

Common Failure Points and Debugging Strategies

Job Not Enqueued

When JobService.create() appears to succeed but jobs never appear in Redis:

• Validate the DTO mapping. The asJobItem() function throws BadRequestException for unknown job names. Check server logs for "Invalid job name" errors.
• Inspect deduplication logic. Jobs with a jobId parameter use BullMQ's add() method, which silently ignores duplicates. Verify unique IDs if resubmitting jobs.
• Query the repository directly:

  const jobs = await jobRepository.searchJobs(
    QueueName.Default,
    { status: [QueueJobStatus.Waiting] }
  );
  console.log(jobs);

Worker Never Picks Up the Job

If jobs sit in "waiting" status indefinitely:

• Verify worker startup. JobRepository.startWorkers() logs each queue initialization. Confirm the expected queue appears in startup logs.
• Check paused state. Call await jobRepository.isPaused(QueueName.Default) or check the Bull Board UI. Paused queues do not feed workers.
• Review concurrency settings. Low concurrency limits can make queues appear stuck under load. Use jobRepository.setConcurrency(queue, 5) to increase parallelism.

Handler Throws or Returns Unexpected Status

When jobs fail or skip without clear errors:

• Trace the event trail. JobService.onJobRun() emits JobStart, then JobSuccess or JobError. Search logs for JobError payloads containing { job, error }.
• Validate handler registration. Missing @OnJob decorators cause JobRepository.run() to return JobStatus.Skipped with a warning log. Verify the decorator includes correct name and queue properties.
• Check return values. Handlers must return JobStatus.Success, JobStatus.Skipped, or JobStatus.Failed. Other return types prevent onDone() from triggering follow-up jobs.

Follow-Up Jobs Not Triggered

When primary jobs succeed but subsequent processing never starts:

• Inspect onDone() logic. The method only schedules follow-ups for specific JobName values in its switch block. New job types require explicit extension of this logic.
• Verify data payload. Follow-ups often depend on item.data.id or similar fields. Missing properties cause early break statements. Add logging before the switch to inspect item.data shape.

Stale or Zombie Jobs in Redis

When Redis contains jobs that never complete or clear:

• Analyze job counts. await jobRepository.getJobCounts(queue) returns states including active, waiting, and completed. Many "active" jobs with no corresponding logs indicate stuck workers.
• Clean manually. Use await jobRepository.clear(queue, QueueCleanType.Completed) to remove finished jobs, or pause and resume the queue to reset worker state during debugging.

Visibility in the UI

Immich includes Bull Board at /admin/jobs (when enabled). This interface displays:

• Waiting/active/completed job counts per queue
• Job IDs and timestamps
• Payload JSON for inspecting data fields

Use Bull Board to verify jobs transition from waiting to active, and to inspect payload structure before handlers execute.

Practical Debugging Code Snippets

List all waiting jobs for a queue

import { QueueName, QueueJobStatus } from 'src/enum';

async function listWaiting(queueRepo: JobRepository) {
  const waiting = await queueRepo.searchJobs(QueueName.Default, {
    status: [QueueJobStatus.Waiting],
  });
  console.table(
    waiting.map((j) => ({
      id: j.id,
      name: j.name,
      ts: new Date(j.timestamp),
    }))
  );
}

Manually trigger a job for testing

await jobService.create({
  name: ManualJobName.MemoryCreate, // maps to JobName.MemoryGenerate
  // additional data fields go here
});

Force a worker to run a job now

await jobRepository.run({
  name: JobName.AssetGenerateThumbnails,
  data: { id: '12345', source: 'upload', notify: true },
});

Debug a failing handler

import { Logger } from '@nestjs/common';

@Injectable()
export class ThumbnailService {
  private readonly log = new Logger(ThumbnailService.name);

  @OnJob({ name: JobName.AssetGenerateThumbnails, queue: QueueName.Default })
  async generateThumbnails(job: { id: string; source: string }) {
    this.log.debug(`Running thumbnail job for ${job.id}`);
    // handler implementation
    return JobStatus.Success;
  }
}

Key Files to Know

File	Purpose	Source
server/src/services/job.service.ts	High-level API for job creation, JobRun event handling, and follow-up job scheduling.	View source
server/src/repositories/job.repository.ts	Registers handlers via @OnJob, starts BullMQ workers, provides queue management utilities.	View source
server/src/decorators/job.decorator.ts	Defines @OnJob metadata used by the repository to map jobs to handlers.	View source
server/src/enum/index.ts	Central definitions for JobName, QueueName, JobStatus, and QueueJobStatus.	View source
server/src/types.ts	Type definitions for JobItem, JobOf, and related queue interfaces.	View source
server/src/repositories/event.repository.ts	Event bus implementation for JobStart, JobSuccess, JobError, and JobComplete emissions.	View source
server/src/repositories/config.repository.ts	Contains BullMQ connection settings via configRepository.getEnv().bull.	View source

Summary

• Trace the enqueue flow: Verify JobService.create() → asJobItem() → JobRepository.queue() to confirm jobs reach Redis.
• Monitor event emissions: Watch for JobStart, JobSuccess, JobError, and JobComplete in logs to pinpoint where execution fails.
• Validate handler registration: Ensure @OnJob decorators are present and return JobStatus.Success to trigger onDone() follow-ups.
• Inspect Redis directly: Use searchJobs(), getJobCounts(), and Bull Board at /admin/jobs to verify job states and payloads.
• Check concurrency and pauses: Low concurrency or paused queues can make workers appear unresponsive despite healthy job counts.

Frequently Asked Questions

Why is my job not showing up in the Bull Board UI?

If a job does not appear in the Bull Board at /admin/jobs, it likely failed validation in asJobItem() or was deduplicated by BullMQ due to a duplicate jobId. Check the server logs for BadRequestException errors immediately after calling JobService.create(), and verify that you are not reusing job IDs when resubmitting.

How do I increase the number of concurrent workers for a specific queue?

Use the setConcurrency() method on JobRepository to adjust parallelism without restarting the server. For example, await jobRepository.setConcurrency(QueueName.Default, 10) increases the worker pool for the default queue to ten concurrent jobs. You can verify the change by monitoring the active job count in Bull Board or via getJobCounts().

What causes a job to be marked as Skipped instead of Success?

A Skipped status occurs when JobRepository.run() cannot find a registered handler for the job name, or when the handler explicitly returns JobStatus.Skipped. Verify that your service method has the @OnJob({ name: JobName.YourJob, queue: QueueName.YourQueue }) decorator and that the method returns JobStatus.Success upon completion. Missing decorators will log a warning in job.repository.ts.

Why are follow-up jobs not triggered after a successful job completion?

Follow-up jobs are scheduled in JobService.onDone(), which only processes specific JobName values in its switch statement. If you added a custom job, you must extend the onDone() logic to queue subsequent work. Additionally, verify that the handler returns JobStatus.Success (not Skipped or an undefined value), as onDone() only triggers on successful completion events.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}