# Debugging Strategies for Immich Background Job Queue Issues: A Complete Guide

> Solve Immich background job queue issues with this guide. Learn debugging strategies by tracing job flow and monitoring events in job.service.ts and job.repository.ts for efficient troubleshooting.

- Repository: [Immich/immich](https://github.com/immich-app/immich)
- Tags: how-to-guide
- Published: 2026-02-27

---

**To debug Immich's background job queue system, trace the flow from `JobService.create()` through `JobRepository.queue()` to the BullMQ workers, monitoring `JobRun` events and handler returns in [`job.service.ts`](https://github.com/immich-app/immich/blob/main/job.service.ts) and [`job.repository.ts`](https://github.com/immich-app/immich/blob/main/job.repository.ts).**

Immich processes long-running tasks—thumbnail generation, facial recognition, and metadata extraction—through a BullMQ-backed job queue orchestrated by `JobService` and `JobRepository`. Understanding the interaction between these two classes is essential for debugging why jobs fail to enqueue, execute, or trigger follow-up work.

## How the Immich Job Queue Works

The queue system follows a strict pipeline from creation to completion:

| Step | Component | Action | Source |
|------|-----------|--------|--------|
| **Enqueue** | `JobService.create()` → `JobRepository.queue()` | Client sends `JobCreateDto`; `asJobItem()` maps manual names to internal `JobName` values. | `job.service.ts:L45-L47` |
| **Storage** | `JobRepository.queueAll()` | Groups jobs by `QueueName` (from `@OnJob` decorator) and adds to BullMQ via `add()` or `addBulk()`. | `job.repository.ts:L60-L84` |
| **Workers** | `JobRepository.startWorkers()` | Instantiates BullMQ `Worker` per queue; each re-emits `JobRun` event for `JobService` to handle. | `job.repository.ts:L87-L96` |
| **Execution** | `JobService.onJobRun()` | Emits `JobStart`, executes handler via `JobRepository.run()`, then emits `JobSuccess` or `JobError`. Always emits `JobComplete`. | `job.service.ts:L49-L63` |
| **Follow-ups** | `JobService.onDone()` | Based on original job name, queues follow-up jobs (e.g., thumbnail generation after upload). | `job.service.ts:L68-L122` |

## Common Failure Points and Debugging Strategies

### Job Not Enqueued

When `JobService.create()` appears to succeed but jobs never appear in Redis:

- **Validate the DTO mapping**. The `asJobItem()` function throws `BadRequestException` for unknown job names. Check server logs for "Invalid job name" errors.
- **Inspect deduplication logic**. Jobs with a `jobId` parameter use BullMQ's `add()` method, which silently ignores duplicates. Verify unique IDs if resubmitting jobs.
- **Query the repository directly**:
  ```typescript
  const jobs = await jobRepository.searchJobs(
    QueueName.Default,
    { status: [QueueJobStatus.Waiting] }
  );
  console.log(jobs);
  ```

### Worker Never Picks Up the Job

If jobs sit in "waiting" status indefinitely:

- **Verify worker startup**. `JobRepository.startWorkers()` logs each queue initialization. Confirm the expected queue appears in startup logs.
- **Check paused state**. Call `await jobRepository.isPaused(QueueName.Default)` or check the Bull Board UI. Paused queues do not feed workers.
- **Review concurrency settings**. Low concurrency limits can make queues appear stuck under load. Use `jobRepository.setConcurrency(queue, 5)` to increase parallelism.

### Handler Throws or Returns Unexpected Status

When jobs fail or skip without clear errors:

- **Trace the event trail**. `JobService.onJobRun()` emits `JobStart`, then `JobSuccess` or `JobError`. Search logs for `JobError` payloads containing `{ job, error }`.
- **Validate handler registration**. Missing `@OnJob` decorators cause `JobRepository.run()` to return `JobStatus.Skipped` with a warning log. Verify the decorator includes correct `name` and `queue` properties.
- **Check return values**. Handlers must return `JobStatus.Success`, `JobStatus.Skipped`, or `JobStatus.Failed`. Other return types prevent `onDone()` from triggering follow-up jobs.

### Follow-Up Jobs Not Triggered

When primary jobs succeed but subsequent processing never starts:

- **Inspect `onDone()` logic**. The method only schedules follow-ups for specific `JobName` values in its `switch` block. New job types require explicit extension of this logic.
- **Verify data payload**. Follow-ups often depend on `item.data.id` or similar fields. Missing properties cause early `break` statements. Add logging before the switch to inspect `item.data` shape.

### Stale or Zombie Jobs in Redis

When Redis contains jobs that never complete or clear:

- **Analyze job counts**. `await jobRepository.getJobCounts(queue)` returns states including `active`, `waiting`, and `completed`. Many "active" jobs with no corresponding logs indicate stuck workers.
- **Clean manually**. Use `await jobRepository.clear(queue, QueueCleanType.Completed)` to remove finished jobs, or pause and resume the queue to reset worker state during debugging.

### Visibility in the UI

Immich includes **Bull Board** at `/admin/jobs` (when enabled). This interface displays:

- **Waiting/active/completed** job counts per queue
- **Job IDs** and **timestamps**
- **Payload JSON** for inspecting data fields

Use Bull Board to verify jobs transition from waiting to active, and to inspect payload structure before handlers execute.

## Practical Debugging Code Snippets

### List all waiting jobs for a queue

```typescript
import { QueueName, QueueJobStatus } from 'src/enum';

async function listWaiting(queueRepo: JobRepository) {
  const waiting = await queueRepo.searchJobs(QueueName.Default, {
    status: [QueueJobStatus.Waiting],
  });
  console.table(
    waiting.map((j) => ({
      id: j.id,
      name: j.name,
      ts: new Date(j.timestamp),
    }))
  );
}

```

### Manually trigger a job for testing

```typescript
await jobService.create({
  name: ManualJobName.MemoryCreate, // maps to JobName.MemoryGenerate
  // additional data fields go here
});

```

### Force a worker to run a job now

```typescript
await jobRepository.run({
  name: JobName.AssetGenerateThumbnails,
  data: { id: '12345', source: 'upload', notify: true },
});

```

### Debug a failing handler

```typescript
import { Logger } from '@nestjs/common';

@Injectable()
export class ThumbnailService {
  private readonly log = new Logger(ThumbnailService.name);

  @OnJob({ name: JobName.AssetGenerateThumbnails, queue: QueueName.Default })
  async generateThumbnails(job: { id: string; source: string }) {
    this.log.debug(`Running thumbnail job for ${job.id}`);
    // handler implementation
    return JobStatus.Success;
  }
}

```

## Key Files to Know

| File | Purpose | Source |
|------|---------|--------|
| [`server/src/services/job.service.ts`](https://github.com/immich-app/immich/blob/main/server/src/services/job.service.ts) | High-level API for job creation, `JobRun` event handling, and follow-up job scheduling. | [View source](https://github.com/immich-app/immich/blob/main/server/src/services/job.service.ts) |
| [`server/src/repositories/job.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/job.repository.ts) | Registers handlers via `@OnJob`, starts BullMQ workers, provides queue management utilities. | [View source](https://github.com/immich-app/immich/blob/main/server/src/repositories/job.repository.ts) |
| [`server/src/decorators/job.decorator.ts`](https://github.com/immich-app/immich/blob/main/server/src/decorators/job.decorator.ts) | Defines `@OnJob` metadata used by the repository to map jobs to handlers. | [View source](https://github.com/immich-app/immich/tree/main/server/src/decorators) |
| [`server/src/enum/index.ts`](https://github.com/immich-app/immich/blob/main/server/src/enum/index.ts) | Central definitions for `JobName`, `QueueName`, `JobStatus`, and `QueueJobStatus`. | [View source](https://github.com/immich-app/immich/blob/main/server/src/enum/index.ts) |
| [`server/src/types.ts`](https://github.com/immich-app/immich/blob/main/server/src/types.ts) | Type definitions for `JobItem`, `JobOf`, and related queue interfaces. | [View source](https://github.com/immich-app/immich/blob/main/server/src/types.ts) |
| [`server/src/repositories/event.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/event.repository.ts) | Event bus implementation for `JobStart`, `JobSuccess`, `JobError`, and `JobComplete` emissions. | [View source](https://github.com/immich-app/immich/blob/main/server/src/repositories/event.repository.ts) |
| [`server/src/repositories/config.repository.ts`](https://github.com/immich-app/immich/blob/main/server/src/repositories/config.repository.ts) | Contains BullMQ connection settings via `configRepository.getEnv().bull`. | [View source](https://github.com/immich-app/immich/blob/main/server/src/repositories/config.repository.ts) |

## Summary

- **Trace the enqueue flow**: Verify `JobService.create()` → `asJobItem()` → `JobRepository.queue()` to confirm jobs reach Redis.
- **Monitor event emissions**: Watch for `JobStart`, `JobSuccess`, `JobError`, and `JobComplete` in logs to pinpoint where execution fails.
- **Validate handler registration**: Ensure `@OnJob` decorators are present and return `JobStatus.Success` to trigger `onDone()` follow-ups.
- **Inspect Redis directly**: Use `searchJobs()`, `getJobCounts()`, and Bull Board at `/admin/jobs` to verify job states and payloads.
- **Check concurrency and pauses**: Low concurrency or paused queues can make workers appear unresponsive despite healthy job counts.

## Frequently Asked Questions

### Why is my job not showing up in the Bull Board UI?

If a job does not appear in the Bull Board at `/admin/jobs`, it likely failed validation in `asJobItem()` or was deduplicated by BullMQ due to a duplicate `jobId`. Check the server logs for `BadRequestException` errors immediately after calling `JobService.create()`, and verify that you are not reusing job IDs when resubmitting.

### How do I increase the number of concurrent workers for a specific queue?

Use the `setConcurrency()` method on `JobRepository` to adjust parallelism without restarting the server. For example, `await jobRepository.setConcurrency(QueueName.Default, 10)` increases the worker pool for the default queue to ten concurrent jobs. You can verify the change by monitoring the `active` job count in Bull Board or via `getJobCounts()`.

### What causes a job to be marked as `Skipped` instead of `Success`?

A `Skipped` status occurs when `JobRepository.run()` cannot find a registered handler for the job name, or when the handler explicitly returns `JobStatus.Skipped`. Verify that your service method has the `@OnJob({ name: JobName.YourJob, queue: QueueName.YourQueue })` decorator and that the method returns `JobStatus.Success` upon completion. Missing decorators will log a warning in [`job.repository.ts`](https://github.com/immich-app/immich/blob/main/job.repository.ts).

### Why are follow-up jobs not triggered after a successful job completion?

Follow-up jobs are scheduled in `JobService.onDone()`, which only processes specific `JobName` values in its switch statement. If you added a custom job, you must extend the `onDone()` logic to queue subsequent work. Additionally, verify that the handler returns `JobStatus.Success` (not `Skipped` or an undefined value), as `onDone()` only triggers on successful completion events.