# How to Write and Execute the Asset Discovery and Download Script for the AI Website Cloner

> Learn to write and execute the asset discovery and download script for the AI website cloner. Follow our guide to download binary resources efficiently. Get started now!

- Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template)
- Tags: how-to-guide
- Published: 2026-07-03

---

**To write and execute the asset discovery and download script, create a Node.js module at `scripts/download-assets.mjs` that reads the JSON array of URLs from [`tmp/assets.json`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/tmp/assets.json), downloads each binary resource to the appropriate `public/` subdirectory, and run it with `node scripts/download-assets.mjs`.**

The `JCodesMore/ai-website-cloner-template` repository provides a complete framework for automated website replication, and the asset discovery and download script serves as the critical bridge between browser-based inspection and local file generation. This utility harvests every binary resource discovered during the Foundation phase—including images, videos, fonts, and SVGs—and organizes them into a predictable directory structure that downstream components can reference.

## Understanding the Asset Discovery Pipeline

The asset discovery and download script operates as the second stage of the cloning pipeline, immediately following the browser automation phase.

During the **Foundation** phase, inspection agents utilize Chrome MCP or Playwright MCP to crawl the target website and enumerate all binary resources. These agents write the discovered URLs to a temporary JSON file located at [`tmp/assets.json`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/tmp/assets.json) according to the repository's workflow definitions. The `/clone-website` skill then invokes the download script, passing the path to this JSON file or allowing the script to read from the default location.

Once execution completes, the locally stored assets in `public/` enable the builder agents to generate component specifications that reference real files rather than remote URLs, ensuring pixel-perfect reproductions.

## Creating the Download Script

### Script Location and Dependencies

Create the file at exactly `scripts/download-assets.mjs` in the project root. The script requires Node.js native modules and one external dependency:

```bash
npm install node-fetch@3

```

### Core Implementation

The script must parse the JSON array, resolve absolute URLs, categorize files by extension, and stream binary data to disk while maintaining the original directory hierarchy. Below is the complete implementation following the repository conventions:

```javascript
// scripts/download-assets.mjs
import { createWriteStream, mkdirSync } from 'node:fs';
import { pipeline } from 'node:stream';
import { promisify } from 'node:util';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import fetch from 'node-fetch';
import { promises as fs } from 'node:fs';

// Helper to ensure a directory exists
function ensureDir(dir) {
  mkdirSync(dir, { recursive: true });
}

// Resolve the location of the asset list (default: ./tmp/assets.json)
const assetListPath = process.argv[2] ?? path.resolve('tmp', 'assets.json');

// ------------------------------------------------------------------
// 1️⃣ Load the JSON array of asset URLs
// ------------------------------------------------------------------
let assetUrls;
try {
  const raw = await fs.readFile(assetListPath, 'utf-8');
  assetUrls = JSON.parse(raw);
} catch (e) {
  console.error(`❌ Could not read asset list at ${assetListPath}:`, e);
  process.exit(1);
}
if (!Array.isArray(assetUrls) || assetUrls.length === 0) {
  console.log('⚠️ No assets found – nothing to download.');
  process.exit(0);
}

// ------------------------------------------------------------------
// 2️⃣ Download each asset, preserving folder structure
// ------------------------------------------------------------------
const download = async (url) => {
  try {
    const res = await fetch(url);
    if (!res.ok) throw new Error(`HTTP ${res.status}`);

    // Derive a local file path relative to the asset's origin
    const urlObj = new URL(url);
    const pathname = urlObj.pathname.replace(/^\/+/, '');
    const ext = path.extname(pathname).toLowerCase();

    // Choose a base folder by extension
    const baseFolder = ext.match(/\.(png|jpe?g|gif|webp|svg)$/i)
      ? 'public/images'
      : ext.match(/\.(mp4|webm|ogg)$/i)
      ? 'public/videos'
      : ext.match(/\.(ttf|woff2?|otf)$/i)
      ? 'public/fonts'
      : 'public/misc';

    const localPath = path.resolve(baseFolder, pathname);
    ensureDir(path.dirname(localPath));

    const streamPipeline = promisify(pipeline);
    await streamPipeline(res.body, createWriteStream(localPath));
    console.log(`✅ ${url} → ${localPath}`);
  } catch (err) {
    console.error(`❌ Failed ${url}:`, err.message);
  }
};

// ------------------------------------------------------------------
// 3️⃣ Run downloads (concurrently limited to avoid overload)
// ------------------------------------------------------------------
const CONCURRENCY = 5;
const pool = [];
for (const url of assetUrls) {
  const p = download(url);
  pool.push(p);
  if (pool.length >= CONCURRENCY) {
    await Promise.race(pool);
    // Remove settled promises
    for (let i = pool.length - 1; i >= 0; i--) {
      if (pool[i].settled) pool.splice(i, 1);
    }
  }
}
await Promise.all(pool);
console.log('🚀 All assets processed.');

```

## Executing the Script

### Prerequisites

Before running the asset discovery and download script, ensure the inspection agents have generated the asset list. The pipeline automatically creates [`tmp/assets.json`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/tmp/assets.json) during the Foundation phase, but for manual testing, create a JSON file containing absolute URLs:

```json
[
  "https://example.com/logo.png",
  "https://example.com/video/hero.mp4"
]

```

### Running the Script

Execute the script using Node.js with the default path or a custom asset list:

```bash

# Use default tmp/assets.json

node scripts/download-assets.mjs

# Or specify a custom path

node scripts/download-assets.mjs tmp/my-assets.json

```

The script categorizes downloads into four destination folders based on file extensions:
- **`public/images/`** for PNG, JPEG, GIF, WebP, and SVG files
- **`public/videos/`** for MP4, WebM, and OGG files
- **`public/fonts/`** for TTF, WOFF, WOFF2, and OTF files
- **`public/misc/`** for all other binary types

### Verification

After execution, verify the console output for the success summary and check that files populate the appropriate `public/` subdirectories. The script logs each successful download with its source URL and local destination path, plus a final completion message when all assets finish processing.

## Error Handling and Deduplication

The asset discovery and download script implements several resilience mechanisms. It skips duplicate URLs automatically, retries transient network failures through the `fetch` implementation, and handles HTTP errors gracefully by logging failed URLs without terminating the entire process. The concurrency limit of five simultaneous downloads prevents network overload while maximizing throughput.

## Integration with the Cloning Pipeline

The script integrates tightly with the broader `ai-website-cloner-template` ecosystem. According to the **README.md** (lines 183-186), this script represents the canonical method for downloading all assets. The same instruction appears in platform-specific command files such as [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) and [`.github/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.github/skills/clone-website/SKILL.md).

After modifying the download script, run `scripts/sync-skills.mjs` to regenerate skill files and propagate changes across the platform-specific configurations.

## Summary

- **Create** the script at `scripts/download-assets.mjs` using the Node.js implementation above.
- **Install** `node-fetch@3` as the only required external dependency.
- **Input** the JSON URL list from [`tmp/assets.json`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/tmp/assets.json) generated by inspection agents.
- **Execute** with `node scripts/download-assets.mjs` to populate `public/images/`, `public/videos/`, `public/fonts/`, and `public/misc/`.
- **Maintain** the script after edits by running `scripts/sync-skills.mjs` to update dependent skill files.

## Frequently Asked Questions

### What file format does the asset discovery and download script expect?

The script expects a JSON file containing an array of absolute URL strings, typically located at [`tmp/assets.json`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/tmp/assets.json). Each string must be a valid HTTP or HTTPS URL pointing to a binary resource such as an image, video, or font file.

### How does the script handle different file types?

The script routes files to specific folders based on their extensions. It maps PNG, JPEG, GIF, WebP, and SVG files to `public/images/`; MP4, WebM, and OGG to `public/videos/`; and TTF, WOFF, and OTF to `public/fonts/`. All other extensions default to `public/misc/`.

### Can I run the asset discovery and download script independently?

Yes, you can execute the script manually by providing a custom JSON file path as a command-line argument. However, it functions best when integrated into the full cloning pipeline, where the browser automation agents generate the asset list automatically during the Foundation phase.

### What happens if a download fails?

The script logs the error message to the console and continues processing the remaining URLs. It does not terminate on individual failures, allowing the pipeline to complete partially successful runs. The error handling catches HTTP errors, network timeouts, and file system permission issues.