How Immich's LibraryService Manages External Library Synchronization and Scanning

The LibraryService orchestrates external library synchronization by acquiring database locks, setting up file system watchers, and managing batch scanning jobs through a multi-phase workflow that handles both real-time changes and periodic full-library scans.

Immich uses external libraries to monitor media folders outside its internal storage structure. The LibraryService, located at server/src/services/library.service.ts, coordinates these operations through a sophisticated architecture that ensures data consistency while efficiently handling file system events.

Phase 1: Initialization and Configuration

During application startup, the service initializes its coordination mechanisms to ensure only one instance manages library operations across a potential cluster of micro-services.

Database Locking and Cron Setup

The onConfigInit method acquires a DatabaseLock.Library lock to guarantee singleton execution. Upon obtaining the lock, it registers a cron job using CronJob.LibraryScan that periodically triggers full-library scans. If real-time watching is enabled in the configuration, the method immediately invokes watchAll() to begin monitoring all configured libraries.

// Simplified initialization flow from onConfigInit
async onConfigInit() {
  // Acquire distributed lock
  await this.databaseRepository.withLock(DatabaseLock.Library, async () => {
    // Register periodic scanning
    this.cronRepository.create({
      name: CronJob.LibraryScan,
      expression: '0 0 * * *', // Daily at midnight
      handler: () => this.handleQueueScanAll()
    });
    
    // Start real-time watchers if enabled
    if (this.config.library.watch) {
      await this.watchAll();
    }
  });
}

Phase 2: Real-time File System Watching

For immediate synchronization, the service leverages file system watchers that react to changes as they occur rather than waiting for scheduled scans.

Chokidar Integration and Event Handling

The watch method creates a chokidar watcher via storageRepository.watch for each library's importPaths. This watcher monitors for add, change, and unlink events while respecting the library's exclusionPatterns.

// Watcher setup in the watch method
async watch(libraryId: string) {
  const library = await this.libraryRepository.get(libraryId);
  
  // Create matcher for supported mime types
  const matcher = new MimeTypeMatcher(
    Object.values(MimeType).filter(m => m.startsWith('image/') || m.startsWith('video/'))
  );
  
  // Initialize chokidar watcher
  const watcher = this.storageRepository.watch(
    library.importPaths,
    {
      ignored: library.exclusionPatterns,
      persistent: true,
      ignoreInitial: true
    }
  );
  
  // Handle file additions and modifications
  watcher.on('add', async (path) => {
    if (matcher.match(path)) {
      await this.jobRepository.queue({
        name: JobName.LibrarySyncFiles,
        data: { libraryId, paths: [path] }
      });
    }
  });
  
  // Handle deletions
  watcher.on('unlink', async (path) => {
    await this.jobRepository.queue({
      name: JobName.LibraryRemoveAsset,
      data: { libraryId, path }
    });
  });
}

Filtering and Job Queuing

The watcher uses a mime-type matcher to process only supported image and video extensions. When events fire, the service queues specific jobs: JobName.LibrarySyncFiles for additions and changes, and JobName.LibraryRemoveAsset for deletions.

Phase 3: Batch Scanning and Asset Synchronization

For comprehensive consistency and initial imports, the service performs batch operations that crawl the entire library structure.

Full Library Scans

The queueScan and queueScanAll methods initiate comprehensive scans. These queue two critical jobs per library: LibrarySyncFilesQueueAll to crawl the disk for new files, and LibrarySyncAssetsQueueAll to verify existing assets against current disk state.

// Triggering a full scan
async queueScan(libraryId: string) {
  await this.jobRepository.queue({
    name: JobName.LibrarySyncFilesQueueAll,
    data: { libraryId }
  });
  await this.jobRepository.queue({
    name: JobName.LibrarySyncAssetsQueueAll,
    data: { libraryId }
  });
}

Disk Crawling and File Discovery

The handleQueueSyncFiles method walks each valid import path using storageRepository.walk. It filters out already-imported files by checking against the database, then queues LibrarySyncFiles jobs containing batches of new file paths for efficient processing.

Asset Import and Database Creation

The handleSyncFiles method processes queued path batches. It creates AssetTable rows through processEntity, which builds database records including checksums, timestamps, and mime types. After bulk insertion, it calls queuePostSyncJobs to schedule sidecar detection and metadata extraction via JobName.SidecarCheck.

Asset Validation and Offline Detection

The handleSyncAssets method reconciles existing database assets with the current file system state. It marks missing files as offline and newly-found files as online, scheduling metadata updates through queuePostSyncJobs where necessary.

Key Helper Methods and Utilities

Several utility methods support the synchronization workflow:

• validateImportPath: Ensures import paths are absolute, exist, are directories, and are readable before processing.
• processEntity: Constructs AssetTable objects with computed checksums, mime types, and timestamps for database insertion.
• queuePostSyncJobs: Schedules follow-up processing including sidecar file detection (JobName.SidecarCheck) and metadata extraction after assets are imported.
• checkExistingAsset: Determines whether existing assets should be marked offline, updated, or ignored based on current file system statistics.

Triggering Scans via the API

Consumers such as API controllers can trigger synchronization through public methods:

// Manually trigger a full scan of a single library
await libraryService.queueScan(libraryId);
// Enqueues LibrarySyncFilesQueueAll & LibrarySyncAssetsQueueAll

// Manually trigger a scan of all external libraries
await libraryService.queueScanAll();
// Queues LibraryScanQueueAll which cascades to every library

Real-time watching requires no explicit API calls once enabled; the watch handlers automatically enqueue LibrarySyncFiles and LibraryRemoveAsset jobs when file system events occur.

Summary

• Singleton Coordination: LibraryService uses DatabaseLock.Library to ensure only one instance manages synchronization across clustered deployments.
• Dual Sync Strategy: Combines real-time file watching via chokidar with periodic batch scans via cron jobs for comprehensive coverage.
• Job-Based Architecture: All operations queue discrete jobs (LibrarySyncFiles, LibrarySyncAssetsQueueAll, etc.) enabling asynchronous processing and failure recovery.
• Path Validation: Strict validation via validateImportPath ensures only accessible, absolute directory paths are processed.
• Asset Lifecycle Management: Handles creation, updates, and offline marking through handleSyncFiles and handleSyncAssets, with automatic metadata extraction scheduling.

Frequently Asked Questions

How does Immich prevent duplicate scans when running multiple server instances?

The LibraryService acquires a distributed DatabaseLock.Library during the onConfigInit lifecycle event. Only the instance that successfully obtains this lock registers the cron job and starts file system watchers. This ensures that even in a horizontally scaled deployment, only one instance manages the synchronization workload.

What is the difference between real-time watching and batch scanning in Immich?

Real-time watching uses chokidar to monitor import paths continuously, immediately queuing sync jobs when files are added, changed, or deleted. Batch scanning runs periodically via cron or manual triggers, crawling the entire library structure to discover new files and validate existing assets against the current disk state. Watching provides low-latency updates for active changes, while scanning ensures comprehensive consistency and handles missed events.

How does the service handle file deletions in external libraries?

When the chokidar watcher detects an unlink event (file deletion), the watch method queues a JobName.LibraryRemoveAsset job containing the library ID and file path. This job processes the deletion by locating the corresponding asset in the database and handling the removal according to Immich's asset lifecycle policies, ensuring the database remains consistent with the file system state.

What happens when a new file is discovered during a full library scan?

The handleQueueSyncFiles method walks the import paths using storageRepository.walk, filtering out files already present in the database. For new files, it batches paths and queues LibrarySyncFiles jobs. The handleSyncFiles processor then creates AssetTable records via processEntity, computing checksums and mime types, bulk-inserts them into the database, and finally calls queuePostSyncJobs to schedule metadata extraction and sidecar file detection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}