What Is the `--no-if-modified-since` Flag in wget?

The --no-if-modified-since flag forces wget to ignore local file timestamps and unconditionally fetch every resource, preventing 304 Not Modified responses that would otherwise skip files.

The Website-downloader project by AhmadIbrahiim uses this flag to guarantee complete website mirrors. Without it, wget might skip downloading files that appear unchanged based on local cache, leading to incomplete archives when the application compresses the downloaded site into a ZIP file.

How --no-if-modified-since Works

By default, wget optimizes bandwidth by checking whether local copies of files already exist. When downloading to a directory that contains files from previous runs, wget sends an If-Modified-Since HTTP header containing the local file's timestamp. If the server determines the file hasn't changed, it returns 304 Not Modified, and wget skips the download.

The --no-if-modified-since option disables this behavior entirely. Wget omits the timestamp header and always requests the full file content, ensuring the server transmits every byte regardless of previous download history.

Why the Website-Downloader Requires This Flag

The application creates a brand-new temporary folder for each download request, but the underlying wget process may still reference cached timestamps from previous runs targeting the same hostname. This creates a critical risk: if wget skips files it deems "unchanged," the resulting download directory will be incomplete.

Since the project subsequently compresses the downloaded folder using archiver, any missing files would propagate into the final ZIP archive. This breaks the user expectation of receiving a complete, offline-ready website copy. The --no-if-modified-since flag eliminates this risk by mandating a fresh fetch of every asset.

Implementation in wget/index.js

The flag is hardcoded into the wget spawn command at line 20 of the source file:

// wget/index.js - Line 20
const child = exec(`wget -mkEpnp --no-if-modified-since ${data.website}`);

This implementation ensures that every invocation of the download function performs an unconditional fetch. The ${data.website} variable injects the target URL, while the preceding flags configure the mirroring behavior.

Complete wget Command Breakdown

The full command used in the repository combines several flags with --no-if-modified-since:

wget -mkEpnp --no-if-modified-since https://example.com

Flag explanations:

• -m (mirror): Enables recursive download with infinite depth and timestamping.
• -k (convert links): Rewrites links in the downloaded HTML to work offline.
• -E ( adjust extensions): Adds .html extensions to files that lack them.
• -p (page requisites): Downloads all images, CSS, and other assets needed to display pages.
• -n (no parent): Prevents wget from ascending to parent directories.
• --no-if-modified-since: Forces fresh downloads by ignoring local timestamps.

Summary

• The --no-if-modified-since flag prevents wget from sending If-Modified-Since headers, eliminating 304 Not Modified responses.
• In the Website-downloader project, this ensures complete file retrieval even when processing sites previously downloaded to the same environment.
• The flag is implemented in wget/index.js at line 20, combined with -mkEpnp for full site mirroring.
• Without this flag, the archiver module would create incomplete ZIP files containing only changed resources.
• This guarantees users receive a complete, offline-ready copy of the target website.

Frequently Asked Questions

What does the --no-if-modified-since flag do in wget?

The --no-if-modified-since flag instructs wget to ignore local file timestamps and omit the If-Modified-Since HTTP header from requests. This forces the server to return the full file content rather than a 304 Not Modified status, ensuring wget downloads every resource regardless of previous download history.

Why is --no-if-modified-since important for website mirroring?

In website mirroring, completeness is critical. If wget skips files based on timestamp comparisons, the local mirror will be missing resources that haven't changed since the last download. For the Website-downloader project, this would result in broken links and missing assets in the final archive, violating the expectation of a complete offline copy.

Does using --no-if-modified-since affect download performance?

Yes, using --no-if-modified-since increases bandwidth usage and download time because wget cannot skip unchanged files. Every request results in a full file transfer from the server. However, this trade-off is necessary for the Website-downloader use case to guarantee archive completeness and avoid caching issues across temporary directories.

Where is the --no-if-modified-since flag implemented in the Website-downloader project?

The flag is hardcoded in the command string at line 20 of wget/index.js in the AhmadIbrahiim/Website-downloader repository. It appears in the execution template: wget -mkEpnp --no-if-modified-since ${data.website}, ensuring every download session fetches all files unconditionally.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}