# How the AI Website-Cloner Template Handles Image and Asset Fetching

> Discover how the AI website cloner template fetches images and assets. Learn about its Node.js script, binary streaming, and parallel fetch limits.

- Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template)
- Tags: deep-dive
- Published: 2026-06-27

---

**The template extracts asset URLs during reconnaissance and downloads them using a Node.js script that streams binary resources into the `public/` directory with a concurrency limit of four parallel fetches.**

The **JCodesMore/ai-website-cloner-template** automates image and asset fetching through a dedicated Foundation step that executes after initial site reconnaissance. This process ensures every visual element—from JPEGs to SVGs—is preserved locally to enable pixel-perfect website replication. By combining automated URL extraction with a purpose-built download utility, the template creates a self-contained asset pipeline that serves cloned content without external dependencies.

## Asset Discovery During the Reconnaissance Phase

### Extracting URLs from Page Elements

During the reconnaissance phase, the AI agent inspects the live target page to gather every visual resource reference. According to the inspection logic documented in [`docs/research/INSPECTION_GUIDE.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/docs/research/INSPECTION_GUIDE.md), the system extracts `<img>` src attributes, `<video>` src tags, CSS `background-image` URLs, and inline `<svg>` elements. These discovered URLs are recorded alongside their computed layout information in extraction output files under `docs/research/`.

### Cataloging Binary Resources

The template categorizes discovered assets by type and original path, creating a structured manifest that feeds into the download step. This catalog includes direct links to binary resources hosted on the target domain, ensuring the subsequent fetch process receives a complete list of required files.

## The Download Assets Implementation

### Node.js Fetch Script

The specification instructs the agent to execute `scripts/download-assets.mjs`, a Node.js utility that processes the URL manifest from the reconnaissance phase. As defined in the Foundation step of [`README.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/README.md), this script utilizes the global `fetch` API (or a lightweight `node-fetch` polyfill when necessary) to stream binary resources. The implementation writes downloaded files to the repository’s `public/` folder while preserving meaningful directory hierarchies such as `public/images/` and `public/videos/`.

### Concurrency and Error Handling

The download script implements a concurrency limit of **four parallel downloads** to prevent overwhelming target servers. It includes retry logic for failed network requests and logs any unsuccessful fetches without halting the entire process. This resilient approach ensures that temporary network errors or missing assets do not prevent the completion of the clone operation.

```typescript
import { writeFile, mkdir } from 'node:fs/promises';
import { resolve, dirname } from 'node:path';
import fetch from 'node-fetch';               // fallback when global fetch is unavailable

// Example input: [{ url: 'https://example.com/img/logo.png', path: 'images/logo.png' }, …]
async function downloadAssets(assets: {url: string; path: string}[]) {
  const concurrency = 4;
  const queue = assets.slice();

  async function worker() {
    while (queue.length) {
      const {url, path} = queue.shift()!;
      try {
        const res = await fetch(url);
        if (!res.ok) throw new Error(`HTTP ${res.status}`);
        const data = await res.arrayBuffer();
        const localPath = resolve('public', path);
        await mkdir(dirname(localPath), {recursive: true});
        await writeFile(localPath, Buffer.from(data));
        console.log(`✔︎ ${url} → ${path}`);
      } catch (e) {
        console.warn(`✗ Failed ${url}:`, e);
      }
    }
  }

  // launch limited‑parallel workers
  await Promise.all(Array.from({length: concurrency}, worker));
}

```

## Storage Structure and Next.js Integration

### Public Directory Organization

Downloaded assets reside in the `public/` directory, with subfolders organized by asset type. The script maintains the original directory structure where semantically meaningful, placing images in `public/images/`, videos in `public/videos/`, and other resources in appropriately named subdirectories. This organization ensures that relative path references remain intuitive and maintainable.

### Static Asset Serving

Because the assets live under `public/`, Next.js automatically serves them as static files without additional configuration. The generated React components reference these resources using relative paths such as `src="/images/logo.png"`, allowing the cloned site to serve the exact same visual content as the original. This zero-config approach leverages Next.js built-in static file serving capabilities.

## Workflow Orchestration

The complete image and asset fetching process is documented in [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md), which specifies the exact expectations for the download script including concurrency limits and error handling protocols. The Foundation step described in [`README.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/README.md) orchestrates the transition from URL extraction to binary download, ensuring that asset fetching occurs only after the reconnaissance phase has cataloged all necessary resources.

## Summary

- **Asset Discovery**: The AI agent extracts all image, video, and CSS background URLs during the reconnaissance phase, storing them in `docs/research/`.
- **Download Script**: `scripts/download-assets.mjs` fetches binary resources using Node.js `fetch` with a concurrency limit of four parallel downloads.
- **Resilient Fetching**: Built-in retry logic and error catching allow the process to continue despite individual download failures.
- **Public Storage**: Assets are organized in `public/images/` and `public/videos/` for automatic Next.js static serving.
- **Path Resolution**: Generated components reference downloaded assets via relative paths like `/images/logo.png`.

## Frequently Asked Questions

### How does the template identify which assets to download?

The template scans the target page during the reconnaissance phase to extract every `<img>` src, `<video>` src, CSS `background-image` URL, and inline SVG element. These URLs are cataloged in the research documentation under `docs/research/` before being passed to the download script.

### What happens if an asset download fails?

The `scripts/download-assets.mjs` utility includes error handling that logs failed fetches to the console but continues processing the remaining queue. The script implements retry logic for network errors, ensuring temporary issues do not prevent the completion of the cloning process.

### Where are the downloaded assets stored?

All downloaded binaries are written to the `public/` directory, organized into subfolders like `public/images/` and `public/videos/`. This structure leverages Next.js automatic static file serving, making assets available at the root path without additional configuration.

### Does the template handle video files and SVGs?

Yes, the asset discovery process captures video sources and inline SVG elements alongside traditional image formats. The download script treats these as binary resources, streaming them to the appropriate subdirectories within `public/` to ensure complete visual fidelity in the cloned site.