Child Process Management in Node.js: Handling wget Execution and Termination in Website-Downloader

The Website-downloader repository isolates wget downloads in a Node.js child process, streams real-time progress via stderr, and ensures complete cleanup on SIGTERM termination.

The AhmadIbrahiim/Website-downloader project demonstrates robust child process management by executing the wget binary in an isolated process while maintaining real-time communication with the client. This architecture prevents the main event loop from blocking during long-running downloads and provides granular control over process lifecycle events, including forced termination and partial file cleanup.

Spawning the wget Process with Node.js exec

When a download request arrives, the application creates a new child process using Node’s exec function from the child_process module. This isolates the external binary execution from the main application thread.

Process Creation and Command Flags

In wget/index.js at line 20, the code constructs the wget command with specific flags and executes it:

const { exec } = require('child_process');

// data.website contains the target URL
const child = exec(`wget -mkEpnp --no-if-modified-since ${data.website}`);

The flags -mkEpnp enable mirroring (-m), convert links for local viewing (-k), and include all necessary files (-Epnp), while --no-if-modified-since ensures fresh downloads regardless of timestamps.

Streaming Download Progress in Real Time

The child process communicates download status back to the main application through its stderr stream, which wget uses for progress reporting.

stderr Data Handling and Socket.io Emission

The implementation in wget/index.js (lines 23-34) attaches a data listener to child.stderr to capture progress updates:

child.stderr.on("data", (chunk) => {
  const responseText = chunk.toString();
  
  // Emit progress to the client via Socket.io
  io.emit(data.token, { progress: responseText });
  
  // Extract hostname from "Resolving..." lines for folder naming
  if (responseText.includes("Resolving")) {
    const websiteName = responseText.split("Resolving")[1].split("...")[0].trim();
    website = websiteName;
  }
});

This approach allows the UI to display real-time download progress while the server simultaneously extracts the resolved hostname from the first "Resolving …" line, storing it in the website variable for later use.

Completion Detection and Archiver Handoff

When the stderr stream closes, indicating the download has finished or terminated, the application determines the resulting folder name and transitions to the archiving phase.

Folder Resolution and Module Handoff

As implemented in wget/index.js (lines 36-46), the code calculates the download directory and signals the client:

child.stderr.on('close', () => {
  // Determine folder name: use extracted website or fallback to URL-based naming
  const websiteFolder = website || data.website.replace(/[^a-z0-9]/gi, '_').toLowerCase();
  
  // Notify client that conversion is beginning
  io.emit(data.token, { progress: "Converting to zip..." });
  
  // Pass control to the archiver module
  archiver(websiteFolder, io, data);
});

The archiver function (located in archiver/index.js) receives the completed download folder and creates a ZIP archive for client delivery.

Graceful Termination and Partial Download Cleanup

The child process management includes explicit handling for forced termination signals to prevent orphaned files from consuming disk space.

Process Exit Monitoring

In wget/index.js (lines 48-53), the code listens for the exit event to detect when the process terminates:

child.on('exit', (code, signal) => {
  if (signal === 'SIGTERM') {
    console.log('Process terminated');
    removePartiallyDownloadedFiles(website);
  }
});

Recursive Directory Removal

The cleanup utility uses fs.rmdir with the recursive option to remove incomplete downloads:

const fs = require('fs');
const path = require('path');

function removePartiallyDownloadedFiles(website) {
  const dir = path.join(__dirname, '../', website);
  
  fs.rmdir(dir, { recursive: true }, (err) => {
    if (err) {
      console.error(`Error while removing partially downloaded files: ${err.message}`);
    }
  });
}

This ensures that any aborted or forcefully terminated downloads leave no stray files in the filesystem.

Summary

• Process isolation: The exec function in wget/index.js spawns wget as a separate child process with specific mirroring flags to prevent main-thread blocking.
• Real-time streaming: Progress updates flow through stderr to Socket.io clients, enabling live UI feedback during downloads.
• Lifecycle management: The application monitors both stderr close events for completion and process exit events for termination signals.
• Resource cleanup: The removePartiallyDownloadedFiles function recursively deletes partial downloads when SIGTERM is detected, preventing disk space leaks.

Frequently Asked Questions

How does the application prevent blocking during wget downloads?

The code uses Node.js child_process.exec to spawn wget in a separate process, keeping the main event loop free to handle other HTTP requests and Socket.io connections while the download proceeds in the background.

What happens if the wget process is forcefully terminated?

When the child process receives a SIGTERM signal, the exit event handler in wget/index.js detects the signal and invokes removePartiallyDownloadedFiles(), which recursively deletes the partially downloaded folder using fs.rmdir with the { recursive: true } option.

How does the system determine the downloaded folder name?

The code extracts the hostname from wget's "Resolving …" output in the stderr stream. If extraction fails, it falls back to a sanitized version of the input URL created by replacing non-alphanumeric characters with underscores.

Why is stderr used instead of stdout for progress monitoring?

Wget writes its progress information, including connection status and download percentages, to stderr rather than stdout. This design allows stdout to remain available for actual file content while the Node.js application captures real-time status updates through the child.stderr stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}