What Is the RunDownloadJob in Project N.O.M.A.D.? Core Architecture Explained

The RunDownloadJob is the central BullMQ background worker that orchestrates the complete lifecycle of external file downloads in Project N.O.M.A.D., handling resumable HTTP transfers, database metadata persistence, legacy file cleanup, and type-specific post-processing.

The RunDownloadJob serves as the single point of truth for acquiring and integrating external resources in Project N.O.M.A.D., an open-source content management system developed by Crosstalk Solutions. Implemented in admin/app/jobs/run_download_job.ts, this TypeScript job class manages everything from low-level network resilience to high-level business logic like ZIM file registration and semantic search indexing.

Resumable Downloading with Progress Tracking

At its foundation, the RunDownloadJob leverages the doResumableDownload utility from admin/app/utils/downloads.ts to execute robust HTTP transfers. This implementation supports resumable downloads that can recover from network interruptions, optional MIME type restrictions, configurable timeouts, and "force-new" re-downloads when content must be refreshed.

Progress reporting occurs in real-time through BullMQ's native job.updateProgress() API. As bytes stream to disk, the job calculates completion percentages and persists them to the queue, enabling frontend components to display live download status bars.

Metadata Persistence and Catalog Synchronization

When callers supply resourceMetadata parameters, the job creates or updates an InstalledResource database entry upon successful download. According to the implementation in admin/app/jobs/run_download_job.ts (lines 40-64), this persistence layer records:

• Version strings for content management
• Collection references (e.g., wikipedia_en)
• Source URLs and local file paths
• File sizes and installation timestamps

This ensures the application's resource catalog remains synchronized with actual filesystem state, preventing orphaned database records.

Automatic Cleanup of Legacy Resources

The job implements intelligent garbage collection when updating existing resources. As implemented in lines 66-76 of admin/app/jobs/run_download_job.ts, when a newer version replaces an existing installation, the worker automatically removes the previous file from the filesystem. This prevents disk space exhaustion from accumulated outdated versions while maintaining atomic replacement semantics.

Type-Specific Post-Download Processing

RunDownloadJob acts as a workflow orchestrator, dispatching specialized handlers based on the filetype parameter:

ZIM File Integration

For ZIM archives (offline Wikipedia and content packages), the job invokes ZimService.downloadRemoteSuccessCallback to register the file with the Kiwix container. Immediately following registration, it dispatches an EmbedFileJob to index the ZIM content for semantic search capabilities (lines 80-93).

Map File Registration

For Map resources, the job calls MapService.downloadRemoteSuccessCallback to register the geographic data for use in the application's mapping interface (lines 94-96).

Job Management and Deduplication

The class provides static utilities for deterministic job orchestration:

• dispatch(): Configures retry policies with exponential backoff, generates deterministic job IDs from URLs, and enqueues new download tasks
• getByUrl(): Queries the BullMQ queue to identify existing jobs for a specific URL, preventing duplicate concurrent downloads of identical resources

This deduplication mechanism ensures that requesting the same resource twice—whether from user actions or system triggers—only creates a single network transfer and database entry.

How to Dispatch and Monitor Downloads

Dispatching a New Download

Controllers or services can initiate downloads through the static dispatch method:

import { RunDownloadJob } from '#jobs/run_download_job'

await RunDownloadJob.dispatch({
  url: 'https://example.com/resource.zim',
  filepath: '/data/zim/resource.zim',
  timeout: 60000,
  filetype: 'zim',
  resourceMetadata: {
    resource_id: 42,
    version: '2024-03',
    collection_ref: 'wikipedia_en',
  },
})

Checking for Existing Jobs

To prevent duplicate operations, verify whether a download is already in progress:

const existingJob = await RunDownloadJob.getByUrl('https://example.com/resource.zim')
if (existingJob) {
  console.log('Download already queued, job id:', existingJob.id)
}

Monitoring Progress in the UI

Frontend applications can subscribe to progress updates stored in BullMQ:

// In a front-end Inertia hook
const { data, error } = useSWR('/api/downloads/jobs/zim', fetcher)
// `data.progress` reflects the percentage set by job.updateProgress(...)

Integration with the Download Ecosystem

The RunDownloadJob operates within a coordinated architecture of specialized modules:

File	Contribution
admin/app/jobs/run_download_job.ts	Core job implementation driving downloads, metadata handling, and cleanup
admin/app/utils/downloads.ts	doResumableDownload helper for streamed, resumable HTTP transfers
admin/app/services/zim_service.ts	ZIM-specific registration logic and callback definitions
admin/app/services/map_service.ts	Map resource registration and UI integration
admin/app/jobs/embed_file_job.ts	Downstream indexing job for ZIM semantic search
admin/types/downloads.ts	TypeScript definitions for RunDownloadJobParams and status enums

Summary

• RunDownloadJob in admin/app/jobs/run_download_job.ts is the authoritative background worker for all external resource acquisition in Project N.O.M.A.D.
• It provides resumable, verified downloading with real-time progress reporting through BullMQ
• The job maintains database consistency by creating or updating InstalledResource records with version and collection metadata
• Automatic cleanup of legacy files prevents disk space leaks when updating to newer versions
• Type-specific callbacks handle ZIM registration (with subsequent EmbedFileJob dispatch) and Map file integration
• Static helpers dispatch() and getByUrl() provide deterministic job IDs and prevent duplicate concurrent downloads

Frequently Asked Questions

How does RunDownloadJob handle interrupted or failed downloads?

The job utilizes the doResumableDownload utility which supports HTTP range requests, allowing transfers to resume from the last received byte rather than restarting from zero. Combined with BullMQ's configurable retry policies implemented in the dispatch() method, the system automatically reattempts failed downloads with exponential backoff until the configured timeout or retry limit is reached.

What is the difference between RunDownloadJob and EmbedFileJob?

RunDownloadJob handles the acquisition phase—downloading the raw file from external sources, verifying integrity, and persisting metadata. EmbedFileJob is a downstream processor specifically for ZIM files that performs content indexing for semantic search after the download completes. RunDownloadJob explicitly dispatches EmbedFileJob for ZIM resources upon successful download completion.

How does the system prevent duplicate downloads of the same URL?

The getByUrl() static method generates deterministic job IDs by hashing the target URL, allowing the system to query the BullMQ queue for existing jobs before creating new ones. When dispatch() detects an active job for the requested URL, it returns the existing job instance rather than enqueueing a duplicate, ensuring only one network transfer occurs regardless of how many times the resource is requested.

Where is download progress stored and how can frontend applications access it?

Progress percentages are stored in BullMQ's job metadata via job.updateProgress() calls within the download loop. Frontend applications typically access this data through API endpoints that query the queue state, or via real-time subscriptions depending on the application's architecture. The progress value reflects the percentage of bytes written relative to the total file size reported by the HTTP response headers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}