How the AI Website-Cloner Template Handles Image and Asset Fetching

The template extracts asset URLs during reconnaissance and downloads them using a Node.js script that streams binary resources into the public/ directory with a concurrency limit of four parallel fetches.

The JCodesMore/ai-website-cloner-template automates image and asset fetching through a dedicated Foundation step that executes after initial site reconnaissance. This process ensures every visual element—from JPEGs to SVGs—is preserved locally to enable pixel-perfect website replication. By combining automated URL extraction with a purpose-built download utility, the template creates a self-contained asset pipeline that serves cloned content without external dependencies.

Asset Discovery During the Reconnaissance Phase

Extracting URLs from Page Elements

During the reconnaissance phase, the AI agent inspects the live target page to gather every visual resource reference. According to the inspection logic documented in docs/research/INSPECTION_GUIDE.md, the system extracts <img> src attributes, <video> src tags, CSS background-image URLs, and inline <svg> elements. These discovered URLs are recorded alongside their computed layout information in extraction output files under docs/research/.

Cataloging Binary Resources

The template categorizes discovered assets by type and original path, creating a structured manifest that feeds into the download step. This catalog includes direct links to binary resources hosted on the target domain, ensuring the subsequent fetch process receives a complete list of required files.

The Download Assets Implementation

Node.js Fetch Script

The specification instructs the agent to execute scripts/download-assets.mjs, a Node.js utility that processes the URL manifest from the reconnaissance phase. As defined in the Foundation step of README.md, this script utilizes the global fetch API (or a lightweight node-fetch polyfill when necessary) to stream binary resources. The implementation writes downloaded files to the repository’s public/ folder while preserving meaningful directory hierarchies such as public/images/ and public/videos/.

Concurrency and Error Handling

The download script implements a concurrency limit of four parallel downloads to prevent overwhelming target servers. It includes retry logic for failed network requests and logs any unsuccessful fetches without halting the entire process. This resilient approach ensures that temporary network errors or missing assets do not prevent the completion of the clone operation.

import { writeFile, mkdir } from 'node:fs/promises';
import { resolve, dirname } from 'node:path';
import fetch from 'node-fetch';               // fallback when global fetch is unavailable

// Example input: [{ url: 'https://example.com/img/logo.png', path: 'images/logo.png' }, …]
async function downloadAssets(assets: {url: string; path: string}[]) {
  const concurrency = 4;
  const queue = assets.slice();

  async function worker() {
    while (queue.length) {
      const {url, path} = queue.shift()!;
      try {
        const res = await fetch(url);
        if (!res.ok) throw new Error(`HTTP ${res.status}`);
        const data = await res.arrayBuffer();
        const localPath = resolve('public', path);
        await mkdir(dirname(localPath), {recursive: true});
        await writeFile(localPath, Buffer.from(data));
        console.log(`✔︎ ${url} → ${path}`);
      } catch (e) {
        console.warn(`✗ Failed ${url}:`, e);
      }
    }
  }

  // launch limited‑parallel workers
  await Promise.all(Array.from({length: concurrency}, worker));
}

Storage Structure and Next.js Integration

Public Directory Organization

Downloaded assets reside in the public/ directory, with subfolders organized by asset type. The script maintains the original directory structure where semantically meaningful, placing images in public/images/, videos in public/videos/, and other resources in appropriately named subdirectories. This organization ensures that relative path references remain intuitive and maintainable.

Static Asset Serving

Because the assets live under public/, Next.js automatically serves them as static files without additional configuration. The generated React components reference these resources using relative paths such as src="/images/logo.png", allowing the cloned site to serve the exact same visual content as the original. This zero-config approach leverages Next.js built-in static file serving capabilities.

Workflow Orchestration

The complete image and asset fetching process is documented in .windsurf/workflows/clone-website.md, which specifies the exact expectations for the download script including concurrency limits and error handling protocols. The Foundation step described in README.md orchestrates the transition from URL extraction to binary download, ensuring that asset fetching occurs only after the reconnaissance phase has cataloged all necessary resources.

Summary

• Asset Discovery: The AI agent extracts all image, video, and CSS background URLs during the reconnaissance phase, storing them in docs/research/.
• Download Script: scripts/download-assets.mjs fetches binary resources using Node.js fetch with a concurrency limit of four parallel downloads.
• Resilient Fetching: Built-in retry logic and error catching allow the process to continue despite individual download failures.
• Public Storage: Assets are organized in public/images/ and public/videos/ for automatic Next.js static serving.
• Path Resolution: Generated components reference downloaded assets via relative paths like /images/logo.png.

Frequently Asked Questions

How does the template identify which assets to download?

The template scans the target page during the reconnaissance phase to extract every <img> src, <video> src, CSS background-image URL, and inline SVG element. These URLs are cataloged in the research documentation under docs/research/ before being passed to the download script.

What happens if an asset download fails?

The scripts/download-assets.mjs utility includes error handling that logs failed fetches to the console but continues processing the remaining queue. The script implements retry logic for network errors, ensuring temporary issues do not prevent the completion of the cloning process.

Where are the downloaded assets stored?

All downloaded binaries are written to the public/ directory, organized into subfolders like public/images/ and public/videos/. This structure leverages Next.js automatic static file serving, making assets available at the root path without additional configuration.

Does the template handle video files and SVGs?

Yes, the asset discovery process captures video sources and inline SVG elements alongside traditional image formats. The download script treats these as binary resources, streaming them to the appropriate subdirectories within public/ to ensure complete visual fidelity in the cloned site.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}