How to Write and Execute the Asset Discovery and Download Script for the AI Website Cloner

To write and execute the asset discovery and download script, create a Node.js module at scripts/download-assets.mjs that reads the JSON array of URLs from tmp/assets.json, downloads each binary resource to the appropriate public/ subdirectory, and run it with node scripts/download-assets.mjs.

The JCodesMore/ai-website-cloner-template repository provides a complete framework for automated website replication, and the asset discovery and download script serves as the critical bridge between browser-based inspection and local file generation. This utility harvests every binary resource discovered during the Foundation phase—including images, videos, fonts, and SVGs—and organizes them into a predictable directory structure that downstream components can reference.

Understanding the Asset Discovery Pipeline

The asset discovery and download script operates as the second stage of the cloning pipeline, immediately following the browser automation phase.

During the Foundation phase, inspection agents utilize Chrome MCP or Playwright MCP to crawl the target website and enumerate all binary resources. These agents write the discovered URLs to a temporary JSON file located at tmp/assets.json according to the repository's workflow definitions. The /clone-website skill then invokes the download script, passing the path to this JSON file or allowing the script to read from the default location.

Once execution completes, the locally stored assets in public/ enable the builder agents to generate component specifications that reference real files rather than remote URLs, ensuring pixel-perfect reproductions.

Creating the Download Script

Script Location and Dependencies

Create the file at exactly scripts/download-assets.mjs in the project root. The script requires Node.js native modules and one external dependency:

npm install node-fetch@3

Core Implementation

The script must parse the JSON array, resolve absolute URLs, categorize files by extension, and stream binary data to disk while maintaining the original directory hierarchy. Below is the complete implementation following the repository conventions:

// scripts/download-assets.mjs
import { createWriteStream, mkdirSync } from 'node:fs';
import { pipeline } from 'node:stream';
import { promisify } from 'node:util';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import fetch from 'node-fetch';
import { promises as fs } from 'node:fs';

// Helper to ensure a directory exists
function ensureDir(dir) {
  mkdirSync(dir, { recursive: true });
}

// Resolve the location of the asset list (default: ./tmp/assets.json)
const assetListPath = process.argv[2] ?? path.resolve('tmp', 'assets.json');

// ------------------------------------------------------------------
// 1️⃣ Load the JSON array of asset URLs
// ------------------------------------------------------------------
let assetUrls;
try {
  const raw = await fs.readFile(assetListPath, 'utf-8');
  assetUrls = JSON.parse(raw);
} catch (e) {
  console.error(`❌ Could not read asset list at ${assetListPath}:`, e);
  process.exit(1);
}
if (!Array.isArray(assetUrls) || assetUrls.length === 0) {
  console.log('⚠️ No assets found – nothing to download.');
  process.exit(0);
}

// ------------------------------------------------------------------
// 2️⃣ Download each asset, preserving folder structure
// ------------------------------------------------------------------
const download = async (url) => {
  try {
    const res = await fetch(url);
    if (!res.ok) throw new Error(`HTTP ${res.status}`);

    // Derive a local file path relative to the asset's origin
    const urlObj = new URL(url);
    const pathname = urlObj.pathname.replace(/^\/+/, '');
    const ext = path.extname(pathname).toLowerCase();

    // Choose a base folder by extension
    const baseFolder = ext.match(/\.(png|jpe?g|gif|webp|svg)$/i)
      ? 'public/images'
      : ext.match(/\.(mp4|webm|ogg)$/i)
      ? 'public/videos'
      : ext.match(/\.(ttf|woff2?|otf)$/i)
      ? 'public/fonts'
      : 'public/misc';

    const localPath = path.resolve(baseFolder, pathname);
    ensureDir(path.dirname(localPath));

    const streamPipeline = promisify(pipeline);
    await streamPipeline(res.body, createWriteStream(localPath));
    console.log(`✅ ${url} → ${localPath}`);
  } catch (err) {
    console.error(`❌ Failed ${url}:`, err.message);
  }
};

// ------------------------------------------------------------------
// 3️⃣ Run downloads (concurrently limited to avoid overload)
// ------------------------------------------------------------------
const CONCURRENCY = 5;
const pool = [];
for (const url of assetUrls) {
  const p = download(url);
  pool.push(p);
  if (pool.length >= CONCURRENCY) {
    await Promise.race(pool);
    // Remove settled promises
    for (let i = pool.length - 1; i >= 0; i--) {
      if (pool[i].settled) pool.splice(i, 1);
    }
  }
}
await Promise.all(pool);
console.log('🚀 All assets processed.');

Executing the Script

Prerequisites

Before running the asset discovery and download script, ensure the inspection agents have generated the asset list. The pipeline automatically creates tmp/assets.json during the Foundation phase, but for manual testing, create a JSON file containing absolute URLs:

[
  "https://example.com/logo.png",
  "https://example.com/video/hero.mp4"
]

Running the Script

Execute the script using Node.js with the default path or a custom asset list:

# Use default tmp/assets.json

node scripts/download-assets.mjs

# Or specify a custom path

node scripts/download-assets.mjs tmp/my-assets.json

The script categorizes downloads into four destination folders based on file extensions:

• public/images/ for PNG, JPEG, GIF, WebP, and SVG files
• public/videos/ for MP4, WebM, and OGG files
• public/fonts/ for TTF, WOFF, WOFF2, and OTF files
• public/misc/ for all other binary types

Verification

After execution, verify the console output for the success summary and check that files populate the appropriate public/ subdirectories. The script logs each successful download with its source URL and local destination path, plus a final completion message when all assets finish processing.

Error Handling and Deduplication

The asset discovery and download script implements several resilience mechanisms. It skips duplicate URLs automatically, retries transient network failures through the fetch implementation, and handles HTTP errors gracefully by logging failed URLs without terminating the entire process. The concurrency limit of five simultaneous downloads prevents network overload while maximizing throughput.

Integration with the Cloning Pipeline

The script integrates tightly with the broader ai-website-cloner-template ecosystem. According to the README.md (lines 183-186), this script represents the canonical method for downloading all assets. The same instruction appears in platform-specific command files such as .windsurf/workflows/clone-website.md and .github/skills/clone-website/SKILL.md.

After modifying the download script, run scripts/sync-skills.mjs to regenerate skill files and propagate changes across the platform-specific configurations.

Summary

• Create the script at scripts/download-assets.mjs using the Node.js implementation above.
• Install node-fetch@3 as the only required external dependency.
• Input the JSON URL list from tmp/assets.json generated by inspection agents.
• Execute with node scripts/download-assets.mjs to populate public/images/, public/videos/, public/fonts/, and public/misc/.
• Maintain the script after edits by running scripts/sync-skills.mjs to update dependent skill files.

Frequently Asked Questions

What file format does the asset discovery and download script expect?

The script expects a JSON file containing an array of absolute URL strings, typically located at tmp/assets.json. Each string must be a valid HTTP or HTTPS URL pointing to a binary resource such as an image, video, or font file.

How does the script handle different file types?

The script routes files to specific folders based on their extensions. It maps PNG, JPEG, GIF, WebP, and SVG files to public/images/; MP4, WebM, and OGG to public/videos/; and TTF, WOFF, and OTF to public/fonts/. All other extensions default to public/misc/.

Can I run the asset discovery and download script independently?

Yes, you can execute the script manually by providing a custom JSON file path as a command-line argument. However, it functions best when integrated into the full cloning pipeline, where the browser automation agents generate the asset list automatically during the Foundation phase.

What happens if a download fails?

The script logs the error message to the console and continues processing the remaining URLs. It does not terminate on individual failures, allowing the pipeline to complete partially successful runs. The error handling catches HTTP errors, network timeouts, and file system permission issues.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}