# How the Website-Downloader Application Manages Temporary Files and Disk Space Cleanup

> Learn how Website-downloader efficiently manages temporary files and disk space cleanup. Discover its strategies for archiving and deletion, ensuring optimal storage.

- Repository: [Ahmed Ibrahim/Website-downloader](https://github.com/AhmadIbrahiim/Website-downloader)
- Tags: internals
- Published: 2026-07-08

---

**The Website-downloader application stores mirrored sites in temporary folders named after the target host, archives them into ZIP files in `public/sites/`, and only deletes those temporary folders when the download process receives a SIGTERM signal, leaving successful downloads on disk until manual or scheduled removal.**

The open-source repository `AhmadIbrahiim/Website-downloader` provides a Node.js application that downloads entire websites using `wget` and packages them for user retrieval. Understanding how this tool handles temporary files and disk space cleanup is critical for maintaining server storage, as the default behavior leaves completed downloads on the filesystem while only cleaning up aborted attempts.

## Temporary File Creation During Site Mirroring

When a user initiates a download via the socket.io interface, the application spawns a `wget` process that mirrors the target site into a local directory.

### Host-Based Folder Naming

In [`wget/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/wget/index.js), the temporary folder name is derived from the requested URL. Lines 37–38 assign the folder name based on the `website` variable (or via the `getWebsiteFolderName` function), creating a directory named after the target host (e.g., `example.com`):

```javascript
// wget/index.js (lines 37-38)
var website = getWebsiteFolderName(website);
// wget process writes to ./website/

```

This folder resides in the application root and serves as the working directory for the `wget` process, accumulating all downloaded assets, HTML files, and subdirectories until the mirroring completes or fails.

## Archiving and Permanent Storage

Once the `wget` process finishes successfully, the application transfers the content from the temporary folder to a permanent archive location.

### Creating ZIP Archives from Temporary Directories

The [`archiver/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/archiver/index.js) module handles the compression. Lines 7–9 configure the output stream to write the final ZIP file to `public/sites/`:

```javascript
// archiver/index.js (lines 7-9)
const output = fs.createWriteStream(__dirname + '/public/sites/' + file + '.zip');
const archive = archiver('zip', {
  zlib: { level: 9 }
});

```

At line 48, the archiver adds the entire temporary directory to the ZIP using `archive.directory('./'+file,false);`, which recursively includes all downloaded content. After the archive finalizes, the ZIP file remains in `public/sites/` for user download, but the source temporary folder in the root directory is not automatically deleted.

## Cleanup Logic for Aborted Downloads

The application implements selective cleanup that only triggers during abnormal termination scenarios.

### SIGTERM Handling and Partial Download Removal

If the user aborts the request or the process is killed, the `child` process emits an `'exit'` event. The handler at lines 48–54 in [`wget/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/wget/index.js) checks for the `SIGTERM` signal and invokes `removePartiallyDownloadedFiles(website)`:

```javascript
// wget/index.js (lines 48-54)
child.on('exit', (code, signal) => {
  if (signal === 'SIGTERM') {
    console.log('Process terminated');
    removePartiallyDownloadedFiles(website);
  }
});

```

### The `removePartiallyDownloadedFiles` Function

The cleanup function, defined at lines 56–64, uses `fs.rmdir` with the `recursive: true` option to delete the entire temporary folder and its contents:

```javascript
// wget/index.js (lines 56-64)
function removePartiallyDownloadedFiles(website) {
  fs.rmdir('./' + website, { recursive: true }, (err) => {
    if (err) {
      console.error(err);
      return;
    }
    console.log('Removed partially downloaded files');
  });
}

```

This prevents orphaned data from accumulating when downloads fail or are cancelled, but it does not execute after a successful download-and-archive cycle.

## Disk Space Management Gaps and Recommendations

The current implementation creates a storage liability: successful downloads leave the temporary folder intact alongside the permanent ZIP archive. Running many downloads causes the root directory to accumulate host-named folders (e.g., `example.com`, `another-site.org`) indefinitely.

### Implementing a Periodic Cleanup Routine

To prevent uncontrolled disk usage, add a scheduled cleanup job that removes temporary folders older than a specified threshold. Place this logic in your server entry file (e.g., [`app.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/app.js)):

```javascript
const fs = require('fs');
const path = require('path');

// Remove temporary download folders older than 24 hours
setInterval(() => {
  const root = path.join(__dirname, '..');
  fs.readdir(root, (err, files) => {
    if (err) return;
    files.forEach(f => {
      const dir = path.join(root, f);
      fs.stat(dir, (e, stats) => {
        if (!e && stats.isDirectory() && Date.now() - stats.mtimeMs > 24 * 60 * 60 * 1000) {
          fs.rmdir(dir, { recursive: true }, err => {
            if (!err) console.log(`Cleaned up old folder: ${f}`);
          });
        }
      });
    });
  });
}, 6 * 60 * 60 * 1000); // Run twice daily

```

Alternatively, configure a system cron job to remove directories older than 24 hours from the application root, or modify [`archiver/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/archiver/index.js) to delete the source folder immediately after confirming the ZIP archive is valid.

## Summary

- **Temporary folders** are created in the application root using the target hostname (e.g., `example.com`) as the directory name in [`wget/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/wget/index.js) lines 37–38.
- **Archiving** occurs via [`archiver/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/archiver/index.js) lines 7–9 and 48, which write ZIP files to `public/sites/` but leave the source folder intact.
- **Cleanup only runs on abort**: The `removePartiallyDownloadedFiles` function at [`wget/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/wget/index.js) lines 56–64 executes only when the `wget` process exits with a SIGTERM signal.
- **Successful downloads persist**: Temporary folders remain on disk after archiving completes, requiring manual deletion or a periodic cleanup routine to prevent disk space exhaustion.

## Frequently Asked Questions

### When does the application delete temporary files?

The application deletes temporary files only when the `wget` process is terminated by a SIGTERM signal. According to the exit handler in [`wget/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/wget/index.js) lines 48–54, this triggers `removePartiallyDownloadedFiles()`, which recursively deletes the host-named folder. Successful downloads do not trigger automatic cleanup.

### Where are the downloaded website archives stored?

Completed archives are stored in the `public/sites/` directory relative to the [`archiver/index.js`](https://github.com/AhmadIbrahiim/Website-downloader/blob/main/archiver/index.js) file. The archiver module uses `fs.createWriteStream` to write ZIP files to this location at lines 7–9, making them available for user download while the temporary source data remains in the application root.

### Why does disk space fill up even after downloads complete?

Disk space fills up because the temporary folder created by `wget` (e.g., `./example.com`) is not deleted after the archiver successfully creates the ZIP file. The `removePartiallyDownloadedFiles` function only runs during SIGTERM aborts, not after normal completion, leaving the original mirrored site data on disk alongside the compressed archive.

### How can I prevent orphaned temporary folders from accumulating?

Implement a periodic cleanup mechanism that deletes folders older than 24 hours from the application root directory, or modify the archiver to delete the source folder after verifying the ZIP archive integrity. The repository does not include automatic cleanup for successful downloads, so external maintenance via cron jobs or the `setInterval` pattern shown above is necessary for production deployments.