How to Download ZIM Files Remotely in Project N.O.M.A.D.

Project N.O.M.A.D. provides a secure, resumable download service that fetches ZIM archives via HTTP, validates them against SSRF and MIME-type checks, and processes them through BullMQ background jobs to make them available to the embedded Kiwix reader.

Project N.O.M.A.D. (Network Offline Media Access Device) by Crosstalk Solutions ships with a complete backend infrastructure for acquiring ZIM files from remote sources. When you need to download ZIM files remotely in Project N.O.M.A.D., the system orchestrates the entire workflow—from URL validation and resumable downloading to container restarts and vector embedding generation—through a typed API layer built on AdonisJS and React.

The Remote ZIM Download Architecture

The download flow follows a strict pipeline across multiple layers of the application stack.

API Endpoint and Validation

The process begins at ZimController.downloadRemote in admin/app/controllers/zim_controller.ts. This controller exposes the POST /zim/download-remote endpoint and immediately validates the payload using assertNotPrivateUrl to prevent SSRF attacks. Once validation passes, it delegates execution to ZimService.downloadRemote.

Service Layer and Job Dispatch

The ZimService class in admin/app/services/zim_service.ts handles the core orchestration. The downloadRemote method extracts the filename from the provided URL, verifies the extension ends with .zim, and checks that no existing job is already downloading the same URL. It then creates a background BullMQ job via RunDownloadJob.dispatch, returning a jobId to the caller for progress tracking.

Background Job Execution

Actual file retrieval happens inside RunDownloadJob.handle located in admin/app/jobs/run_download_job.ts. This job performs a resumable HTTP download using doResumableDownload, streaming bytes to disk while updating BullMQ progress percentages. Upon completion, the job:

• Creates or updates an InstalledResource record to register the file locally
• Invokes ZimService.downloadRemoteSuccessCallback to restart the Kiwix container and purge outdated Wikipedia ZIM files
• Dispatches an EmbedFileJob to generate vector embeddings for the new content

How to Trigger a Remote ZIM Download

You can initiate downloads through three interfaces depending on your integration needs.

Using the JavaScript API Client

The front-end wrapper in admin/inertia/lib/api.ts exposes downloadRemoteZimFile, which posts to the endpoint with proper TypeScript types.

import api from '~/lib/api'

// Trigger download from a React component or Inertia page
const { filename, jobId } = await api.downloadRemoteZimFile(
  'https://download.kiwix.org/zim/wikipedia_en_all_maxi_2023-12.zim',
  {
    title: 'Wikipedia – English (Full)',
    author: 'Kiwix',
    size_bytes: 9_123_456_789, // Optional: for UI progress display
  }
)

Using cURL for Automation

For scripts or CI pipelines, hit the endpoint directly with a JSON payload.

curl -X POST http://localhost:3333/api/zim/download-remote \
     -H "Content-Type: application/json" \
     -d '{
           "url":"https://download.kiwix.org/zim/wikipedia_en_all_maxi_2023-12.zim",
           "metadata":{"title":"Wikipedia – English (Full)","author":"Kiwix"}
         }'

The response includes the dispatched job ID:

{
  "message": "Download started successfully",
  "filename": "wikipedia_en_all_maxi_2023-12.zim",
  "jobId": "a1b2c3d4e5f6",
  "url": "https://download.kiwix.org/zim/wikipedia_en_all_maxi_2023-12.zim"
}

Programmatic Access via Node.js

On the backend, instantiate the service directly within an Adonis context.

import ZimService from '#services/zim_service'
import DockerService from '#services/docker_service'

const zimService = new ZimService(new DockerService())
const result = await zimService.downloadRemote(
  'https://download.kiwix.org/zim/wikipedia_en_all_maxi_2023-12.zim'
)
// Returns: { filename, jobId }

Security and Validation Features

The system implements multiple guards to ensure safe remote fetching. The assertNotPrivateUrl validator blocks requests to internal network addresses, preventing SSRF attacks against the host infrastructure. Additionally, the whitelist constant ZIM_MIME_TYPES ensures only legitimate ZIM content types are accepted. The resumable download logic inspects the Content-Length header and respects configurable timeouts to prevent resource exhaustion.

Monitoring Download Progress

The React front-end consumes download status via the useDownloads hook with the parameter {filetype:'zim'}, which polls /downloads/jobs/zim. As RunDownloadJob updates its progress percentage in BullMQ, the UI reflects real-time completion status. When the job reaches 100%, the Kiwix service automatically detects the new file through the InstalledResource record and makes it available for offline reading.

Summary

• Endpoint: Send a POST request to /zim/download-remote via api.downloadRemoteZimFile or direct HTTP calls to download ZIM files remotely in Project N.O.M.A.D.
• Validation: URLs are checked against private IP ranges and must end with the .zim extension to pass ZimService validation.
• Background Processing: Downloads run inside RunDownloadJob using resumable streams, with automatic Kiwix container restarts and embedding generation upon completion.
• Monitoring: Track progress using the useDownloads React Query hook or by querying the BullMQ job status directly.

Frequently Asked Questions

What URL formats are supported for remote ZIM downloads?

The system accepts any public HTTP or HTTPS URL that points to a file ending in .zim. The ZimService.downloadRemote method explicitly checks the URL extension and validates the MIME type against the ZIM_MIME_TYPES whitelist after headers are received.

How does Project N.O.M.A.D. handle interrupted downloads?

The RunDownloadJob class implements doResumableDownload, which supports partial content retrieval via HTTP range requests. If a download is interrupted, the job can resume from the last received byte rather than restarting from the beginning, and progress is persisted in BullMQ.

What security measures protect against malicious URLs?

The ZimController invokes assertNotPrivateUrl to block SSRF attempts against internal networks (localhost, 169.254.x.x, etc.). Additionally, the service verifies the Content-Type header against known ZIM MIME types before writing bytes to disk.

How can I check the status of an active ZIM download?

Query the /downloads/jobs/zim endpoint or use the front-end hook useDownloads({filetype:'zim'}) as implemented in admin/inertia/pages/settings/zim/remote-explorer.tsx. This returns the job list including progress percentages and status fields updated by the background worker.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}