# What Is the RunDownloadJob in Project N.O.M.A.D.? Core Architecture Explained

> Explore the RunDownloadJob in Project N.O.M.A.D. Understand how this core BullMQ worker manages file downloads, persistence, cleanup, and post-processing for efficient data management.

- Repository: [Crosstalk Solutions/project-nomad](https://github.com/Crosstalk-Solutions/project-nomad)
- Tags: architecture
- Published: 2026-03-16

---

**The RunDownloadJob is the central BullMQ background worker that orchestrates the complete lifecycle of external file downloads in Project N.O.M.A.D., handling resumable HTTP transfers, database metadata persistence, legacy file cleanup, and type-specific post-processing.**

The RunDownloadJob serves as the single point of truth for acquiring and integrating external resources in Project N.O.M.A.D., an open-source content management system developed by Crosstalk Solutions. Implemented in [`admin/app/jobs/run_download_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/run_download_job.ts), this TypeScript job class manages everything from low-level network resilience to high-level business logic like ZIM file registration and semantic search indexing.

## Resumable Downloading with Progress Tracking

At its foundation, the RunDownloadJob leverages the `doResumableDownload` utility from [`admin/app/utils/downloads.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/downloads.ts) to execute robust HTTP transfers. This implementation supports **resumable downloads** that can recover from network interruptions, optional MIME type restrictions, configurable timeouts, and "force-new" re-downloads when content must be refreshed.

Progress reporting occurs in real-time through BullMQ's native `job.updateProgress()` API. As bytes stream to disk, the job calculates completion percentages and persists them to the queue, enabling frontend components to display live download status bars.

## Metadata Persistence and Catalog Synchronization

When callers supply `resourceMetadata` parameters, the job creates or updates an `InstalledResource` database entry upon successful download. According to the implementation in [`admin/app/jobs/run_download_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/run_download_job.ts) (lines 40-64), this persistence layer records:

- **Version strings** for content management
- **Collection references** (e.g., `wikipedia_en`)
- **Source URLs** and local file paths
- **File sizes** and installation timestamps

This ensures the application's resource catalog remains synchronized with actual filesystem state, preventing orphaned database records.

## Automatic Cleanup of Legacy Resources

The job implements intelligent garbage collection when updating existing resources. As implemented in lines 66-76 of [`admin/app/jobs/run_download_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/run_download_job.ts), when a newer version replaces an existing installation, the worker automatically removes the previous file from the filesystem. This prevents disk space exhaustion from accumulated outdated versions while maintaining atomic replacement semantics.

## Type-Specific Post-Download Processing

RunDownloadJob acts as a workflow orchestrator, dispatching specialized handlers based on the `filetype` parameter:

### ZIM File Integration

For **ZIM** archives (offline Wikipedia and content packages), the job invokes `ZimService.downloadRemoteSuccessCallback` to register the file with the Kiwix container. Immediately following registration, it dispatches an **EmbedFileJob** to index the ZIM content for semantic search capabilities (lines 80-93).

### Map File Registration

For **Map** resources, the job calls `MapService.downloadRemoteSuccessCallback` to register the geographic data for use in the application's mapping interface (lines 94-96).

## Job Management and Deduplication

The class provides static utilities for deterministic job orchestration:

- **`dispatch()`**: Configures retry policies with exponential backoff, generates deterministic job IDs from URLs, and enqueues new download tasks
- **`getByUrl()`**: Queries the BullMQ queue to identify existing jobs for a specific URL, preventing duplicate concurrent downloads of identical resources

This deduplication mechanism ensures that requesting the same resource twice—whether from user actions or system triggers—only creates a single network transfer and database entry.

## How to Dispatch and Monitor Downloads

### Dispatching a New Download

Controllers or services can initiate downloads through the static dispatch method:

```typescript
import { RunDownloadJob } from '#jobs/run_download_job'

await RunDownloadJob.dispatch({
  url: 'https://example.com/resource.zim',
  filepath: '/data/zim/resource.zim',
  timeout: 60000,
  filetype: 'zim',
  resourceMetadata: {
    resource_id: 42,
    version: '2024-03',
    collection_ref: 'wikipedia_en',
  },
})

```

### Checking for Existing Jobs

To prevent duplicate operations, verify whether a download is already in progress:

```typescript
const existingJob = await RunDownloadJob.getByUrl('https://example.com/resource.zim')
if (existingJob) {
  console.log('Download already queued, job id:', existingJob.id)
}

```

### Monitoring Progress in the UI

Frontend applications can subscribe to progress updates stored in BullMQ:

```typescript
// In a front-end Inertia hook
const { data, error } = useSWR('/api/downloads/jobs/zim', fetcher)
// `data.progress` reflects the percentage set by job.updateProgress(...)

```

## Integration with the Download Ecosystem

The RunDownloadJob operates within a coordinated architecture of specialized modules:

| File | Contribution |
|------|--------------|
| **[`admin/app/jobs/run_download_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/run_download_job.ts)** | Core job implementation driving downloads, metadata handling, and cleanup |
| **[`admin/app/utils/downloads.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/utils/downloads.ts)** | `doResumableDownload` helper for streamed, resumable HTTP transfers |
| **[`admin/app/services/zim_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/zim_service.ts)** | ZIM-specific registration logic and callback definitions |
| **[`admin/app/services/map_service.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/services/map_service.ts)** | Map resource registration and UI integration |
| **[`admin/app/jobs/embed_file_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/embed_file_job.ts)** | Downstream indexing job for ZIM semantic search |
| **[`admin/types/downloads.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/types/downloads.ts)** | TypeScript definitions for `RunDownloadJobParams` and status enums |

## Summary

- **RunDownloadJob** in [`admin/app/jobs/run_download_job.ts`](https://github.com/Crosstalk-Solutions/project-nomad/blob/main/admin/app/jobs/run_download_job.ts) is the authoritative background worker for all external resource acquisition in Project N.O.M.A.D.
- It provides **resumable, verified downloading** with real-time progress reporting through BullMQ
- The job maintains database consistency by creating or updating **InstalledResource** records with version and collection metadata
- Automatic **cleanup of legacy files** prevents disk space leaks when updating to newer versions
- **Type-specific callbacks** handle ZIM registration (with subsequent EmbedFileJob dispatch) and Map file integration
- Static helpers **`dispatch()`** and **`getByUrl()`** provide deterministic job IDs and prevent duplicate concurrent downloads

## Frequently Asked Questions

### How does RunDownloadJob handle interrupted or failed downloads?

The job utilizes the `doResumableDownload` utility which supports HTTP range requests, allowing transfers to resume from the last received byte rather than restarting from zero. Combined with BullMQ's configurable retry policies implemented in the `dispatch()` method, the system automatically reattempts failed downloads with exponential backoff until the configured timeout or retry limit is reached.

### What is the difference between RunDownloadJob and EmbedFileJob?

**RunDownloadJob** handles the acquisition phase—downloading the raw file from external sources, verifying integrity, and persisting metadata. **EmbedFileJob** is a downstream processor specifically for ZIM files that performs content indexing for semantic search after the download completes. RunDownloadJob explicitly dispatches EmbedFileJob for ZIM resources upon successful download completion.

### How does the system prevent duplicate downloads of the same URL?

The `getByUrl()` static method generates deterministic job IDs by hashing the target URL, allowing the system to query the BullMQ queue for existing jobs before creating new ones. When `dispatch()` detects an active job for the requested URL, it returns the existing job instance rather than enqueueing a duplicate, ensuring only one network transfer occurs regardless of how many times the resource is requested.

### Where is download progress stored and how can frontend applications access it?

Progress percentages are stored in BullMQ's job metadata via `job.updateProgress()` calls within the download loop. Frontend applications typically access this data through API endpoints that query the queue state, or via real-time subscriptions depending on the application's architecture. The progress value reflects the percentage of bytes written relative to the total file size reported by the HTTP response headers.