What Wget Flags Are Used for Website Downloading and How They Work Together

The wget -mkEpnp --no-if-modified-since command combines mirroring, link conversion, extension adjustment, and asset downloading to create a fully self-contained offline copy of any website.

The Website-downloader repository by AhmadIbrahiim automates full-site archiving using a carefully selected combination of wget flags. These flags work in concert to recursively download content, adjust internal links for offline browsing, and ensure all visual assets are captured. Understanding this specific flag combination reveals how the tool produces ready-to-use offline websites without broken links or missing resources.

The Complete Wget Command Structure

The project constructs its download command in wget/index.js, executing a single wget invocation with six specific flags. As documented in the project's README.md, the command format is:

wget -mkEpnp --no-if-modified-since https://example.org

This shorthand combines multiple long-form options into a compact instruction set that handles everything from recursive crawling to file extension management.

Breakdown of Each Wget Flag

Each flag serves a distinct purpose in creating a complete offline mirror.

-m (--mirror)

The -m flag enables --mirror mode, which activates recursive downloading with infinite depth, timestamping, and proper file retention. This makes wget crawl the entire site structure starting from the provided URL, grabbing every reachable page and maintaining the directory hierarchy.

-k (--convert-links)

-k or --convert-links rewrites all internal URLs in downloaded HTML and CSS files to point to local file paths rather than original web addresses. This ensures that when you open the downloaded site in a browser, navigation links work correctly without attempting to reach the live internet.

-E (--adjust-extension)

The -E flag (--adjust-extension) automatically appends appropriate file extensions (.html, .css, etc.) based on content MIME types. This prevents files from being saved without extensions, which would otherwise break local browser rendering when opening them offline.

-p (--page-requisites)

-p (--page-requisites) downloads all assets necessary to display each page correctly, including images, stylesheets, JavaScript files, and embedded media. This flag ensures the visual layout and interactive functionality remain intact without requiring external network requests.

-np (--no-parent)

-np or --no-parent restricts the crawler to the specified directory subtree, preventing wget from climbing to parent directories. This protects against accidental over-downloading and keeps the archive focused on the intended target path rather than the entire domain.

--no-if-modified-since

The --no-if-modified-since flag forces wget to ignore cached headers and download fresh copies of every resource. This guarantees that repeated runs of the command fetch updated content rather than preserving potentially stale versions from previous downloads.

How the Flags Work Together

When executed together, these flags create a six-step pipeline:

• Mirror (-m) recursively traverses the entire site structure
• Confine (-np) keeps the crawl within the target directory
• Collect (-p) gathers all required page assets
• Name (-E) ensures files have proper browser-recognized extensions
• Rewrite (-k) converts all URLs to local file references
• Refresh (--no-if-modified-since) guarantees current content on each execution

The result is a directory structure that mirrors the original site exactly, with all internal references self-contained and all assets present for immediate offline use.

Implementation in the Source Code

In wget/index.js, the command is constructed dynamically using JavaScript's exec function:

const child = exec(`wget -mkEpnp --no-if-modified-since ${data.website}`);

This implementation passes the target URL directly into the flag string, executing the complete download sequence. After wget completes, the archiver/index.js module packages the downloaded directory into a ZIP file for distribution.

Summary

• The wget -mkEpnp --no-if-modified-since command provides the complete flag set needed for full-site downloading
• -m enables recursive mirroring while -np constrains the crawl to the target subtree
• -p downloads all page assets and -E ensures proper file extensions
• -k rewrites links for offline browsing and --no-if-modified-since forces fresh downloads
• These flags are implemented in wget/index.js and documented in README.md of the Website-downloader repository

Frequently Asked Questions

What does the wget -mkEpnp flag combination do?

The -mkEpnp combination is shorthand for --mirror --convert-links --adjust-extension --page-requisites --no-parent. Together, these flags create a complete offline copy of a website by recursively downloading all content, adjusting links to work locally, and saving files with proper extensions.

Why is --no-if-modified-since included in the Website-downloader command?

The --no-if-modified-since flag ensures that every execution downloads fresh content rather than relying on cached versions. This prevents stale files when re-downloading sites that may have been updated since the previous archive was created.

Where is the wget command constructed in the Website-downloader repository?

The command is built and executed in wget/index.js using Node.js's exec function, which spawns a child process with the complete flag string and target URL. The README.md file documents the purpose of each flag for reference.

How does the tool prevent downloading the entire domain?

The -np (--no-parent) flag restricts wget to the specified directory and its subdirectories, preventing the crawler from ascending to parent directories or downloading unrelated sections of the domain. This keeps the archive focused and manageable in size.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}