Performance Bottlenecks in Immich's AssetService During Large-Scale Uploads and Indexing

The AssetService in Immich processes uploads and indexing sequentially, creating N+1 database queries, flooding the job queue with individual tasks, and blocking on per-asset filesystem I/O, which becomes a scalability choke point during bulk operations.

asset.service.ts serves as the core orchestration layer in the Immich photo backup solution, coordinating database writes, storage operations, and background job scheduling. When handling large-scale uploads or massive indexing jobs, several sequential processing patterns in this service create significant performance bottlenecks that limit throughput and increase memory pressure.

Sequential Database Writes and N+1 Query Patterns

The primary scalability limitation stems from sequential database updates performed inside asynchronous loops. Methods such as updateAll, deleteAll, and copy invoke repository functions like updateAllExif, updateAll, and update within a single await loop, causing each asset to trigger its own SQL statement.

In server/src/services/asset.service.ts, the updateAll method (lines 61‑89) demonstrates this pattern by awaiting separate repository calls for EXIF data and asset records:

await this.assetRepository.updateAllExif(ids, exifDto);
await this.assetRepository.updateAll(ids, assetDto);
await this.jobRepository.queueAll(ids.map(id => ({ name: JobName.SidecarWrite, data: { id } })));

This N+1 write pattern forces the database to process thousands of individual transactions rather than executing bulk operations, saturating connection pools and increasing latency linearly with asset count.

Per-Asset Job Queue Flooding

After every database mutation, AssetService immediately enqueues individual background jobs. The updateExif method (lines 34‑44) queues a JobName.SidecarWrite for each asset, while the run method (lines 76‑100) creates separate jobs for AssetDetectFaces and AssetExtractMetadata on a per-asset basis.

When hundreds of assets upload simultaneously, this pattern floods the job queue with thousands of discrete tasks. According to the Immich source code, the jobRepository.queueAll call maps every asset ID to an individual job object, increasing queue latency and memory pressure on the Redis or BullMQ backend.

Synchronous Streaming and I/O Blocking

The handleAssetDeletionCheck method (lines 96‑115) implements a streaming pattern that limits parallelism. It iterates through assets using for await, accumulates a chunk into an in-memory array, then awaits queueChunk() before continuing:

for await (const asset of this.assetJobRepository.streamForDeletedJob(trashedBefore)) {
  chunk.push(asset);
  if (chunk.length >= JOBS_ASSET_PAGINATION_SIZE) {
    await this.queueChunk(chunk);
    chunk = [];
  }
}

The await inside the streaming loop forces sequential processing of batches, preventing the next chunk from being prepared while the previous one queues. Additionally, filesystem operations in copySidecar (lines 70‑80) execute storageRepository.copyFile and unlink calls sequentially for each asset, making disk I/O the bottleneck when processing large files.

Memory Pressure and Repeated Traversals

AssetService maintains an in-memory chunk array during deletion checks that grows up to JOBS_ASSET_PAGINATION_SIZE (defined in server/src/constants.ts). During massive purge operations, this buffering can consume significant RAM.

Utility functions compound the CPU overhead. The copySidecar method calls getAssetFiles twice (lines 69‑75) for every asset processed, each invocation iterating over the asset's file list to locate thumbnails, previews, and sidecars. This repeated traversal adds unnecessary CPU cycles when processing thousands of assets.

Optimization Strategies and Refactoring Approaches

Batching Database Operations with Transactions

Replace sequential updates with transactional batch operations. The assetRepository supports bulk methods that can be wrapped in a single transaction:

await this.assetRepository.transaction(async trx => {
  if (Object.keys(exifDto).length) {
    await trx.asset.updateAllExif(ids, exifDto);
  }
  if (Object.keys(assetDto).length) {
    await trx.asset.updateAll(ids, assetDto);
  }
});

Implementing Bulk Job Enqueuing

Instead of mapping individual jobs, implement bulk job handlers that process arrays of asset IDs:

await this.jobRepository.queue({
  name: JobName.SidecarWriteBatch,
  data: { assetIds: ids }
});

Parallelizing Filesystem Operations

Use Promise.allSettled with a concurrency semaphore to prevent blocking on disk I/O:

const MAX_CONCURRENCY = 10;
const sem = new Semaphore(MAX_CONCURRENCY);

await Promise.allSettled(
  assets.map(async asset => {
    await sem.acquire();
    try {
      await this.copySidecar({ sourceAsset: asset.source, targetAsset: asset.target });
    } finally {
      sem.release();
    }
  })
);

Streaming Without In-Memory Accumulation

Eliminate the chunk array by streaming directly to the queue:

for await (const asset of this.assetJobRepository.streamForDeletedJob(trashedBefore)) {
  await this.jobRepository.queue({
    name: JobName.AssetDelete,
    data: { id: asset.id, deleteOnDisk: !asset.isOffline }
  });
}

Summary

• Sequential DB updates in updateAll and deleteAll create N+1 query patterns that saturate database connections during bulk operations.
• Per-asset job queuing in methods like run and updateExif floods the background queue with individual tasks, increasing latency and memory usage.
• Synchronous streaming in handleAssetDeletionCheck awaits each chunk sequentially, limiting throughput during large-scale deletion checks.
• Filesystem I/O operations in copySidecar execute sequentially per asset, creating bottlenecks when copying or deleting large media libraries.
• In-memory buffering using the chunk array and repeated getAssetFiles traversals consume RAM and CPU cycles unnecessarily.

Frequently Asked Questions

Why does AssetService create N+1 database queries during bulk updates?

The updateAll, deleteAll, and copy methods iterate through asset collections and await individual repository calls (updateAllExif, updateAll) for each asset rather than executing a single bulk SQL statement. As implemented in immich-app/immich, this sequential awaiting causes the database to process thousands of separate transactions, creating the classic N+1 write bottleneck that limits throughput during large-scale uploads.

How does per-asset job queuing impact system performance?

Methods like updateExif and run call jobRepository.queueAll with a mapped array of individual job objects (e.g., { name: JobName.SidecarWrite, data: { id } } for every asset). When processing thousands of assets, this pattern injects thousands of discrete tasks into the BullMQ or Redis queue simultaneously, increasing queue depth, memory consumption, and worker polling latency before jobs can be processed.

What causes memory spikes during asset deletion checks?

The handleAssetDeletionCheck method accumulates assets into a chunk array until it reaches JOBS_ASSET_PAGINATION_SIZE (defined in server/src/constants.ts). During massive library purges, this array holds thousands of asset objects in memory before queuing. Combined with repeated calls to getAssetFiles that traverse file lists for each asset, this creates significant RAM pressure and CPU overhead.

Can these bottlenecks be resolved without major architectural changes?

Yes. The asset.service.ts implementation can be optimized by wrapping repository calls in database transactions for true bulk updates, implementing batch job handlers (e.g., SidecarWriteBatch) that accept arrays of IDs, using Promise.allSettled with concurrency limits for filesystem operations, and eliminating the chunk accumulator in favor of direct streaming to the job queue. These changes maintain the existing service architecture while removing sequential blocking patterns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}