How to Automate Downloading and Organizing Assets from Target Websites with the AI Website Cloner Template

The AI Website Cloner Template automates asset acquisition by generating a Node.js script (scripts/download-assets.mjs) that parallel-downloads images, videos, and fonts from the target site into a structured public/ directory, enabling pixel-perfect Next.js rebuilds.

The AI Website Cloner Template converts any live website into a fully functional Next.js application. A critical phase in this pipeline is the automated extraction and organization of binary assets—images, videos, SVGs, and fonts—from the target URL into the project's public/ folder. This process ensures that the cloned site maintains visual fidelity without relying on external CDN links.

Asset Download Automation Architecture

Pipeline Phase Integration

According to AGENTS.md (line 57), the cloning workflow includes a mandatory "Download global assets" step that occurs after visual token extraction but before component generation. This step writes and executes scripts/download-assets.mjs to fetch all external resources enumerated by the AI agent during the Reconnaissance phase.

Directory Structure and Routing

The template enforces strict file-type routing under public/ to mirror the source site’s asset organization:

• Images (PNG, JPG, SVG, WEBP, GIF) → public/images/
• Videos (MP4, WEBM, OGG) → public/videos/
• Fonts (WOFF, WOFF2, TTF, OTF) → public/fonts/
• Miscellaneous binaries → public/misc/

Each asset preserves its remote directory structure under its type folder, mapping https://example.com/assets/img/logo.png to public/images/example.com/assets/img/logo.png.

Implementing the Asset Download Script

While the repository references scripts/download-assets.mjs in documentation—notably at .windsurf/workflows/clone-website.md (line 183) and AGENTS.md (line 57)—the file is generated dynamically during the cloning process. The script implements four core design principles that ensure efficient and reliable asset acquisition:

1. Batch parallelism – Concurrent downloads limited to 4 connections to prevent network saturation
2. Error resilience – Non-blocking failure handling with warning logs for 404s or timeouts
3. Path preservation – Replication of remote URL paths under the local public/ root
4. Type-based routing – Automatic categorization into images/, videos/, or fonts/ subdirectories

Complete Script Implementation

// scripts/download-assets.mjs
import { createWriteStream, mkdirSync } from 'node:fs';
import { pipeline } from 'node:stream';
import { promisify } from 'node:util';
import { URL } from 'node:url';
import fetch from 'node-fetch';

const streamPipeline = promisify(pipeline);
const CONCURRENCY = 4;

// Asset list generated by the AI agent from DOM analysis
const assets = [
  'https://example.com/images/hero.jpg',
  'https://example.com/videos/intro.mp4',
  'https://example.com/fonts/inter.woff2'
];

function getLocalPath(urlStr) {
  const url = new URL(urlStr);
  const pathname = url.pathname.replace(/^\/+/, '');
  const ext = pathname.split('.').pop().toLowerCase();

  let baseDir = 'public';
  if (['png', 'jpg', 'jpeg', 'webp', 'svg', 'gif'].includes(ext)) baseDir += '/images';
  else if (['mp4', 'webm', 'ogg'].includes(ext)) baseDir += '/videos';
  else if (['woff', 'woff2', 'ttf', 'otf'].includes(ext)) baseDir += '/fonts';
  else baseDir += '/misc';

  const fullPath = `${baseDir}/${url.host}/${pathname}`;
  mkdirSync(fullPath.substring(0, fullPath.lastIndexOf('/')), { recursive: true });
  return fullPath;
}

async function download(url) {
  try {
    const res = await fetch(url);
    if (!res.ok) {
      console.warn(`✗ Failed ${url}: ${res.status}`);
      return;
    }
    const dest = createWriteStream(getLocalPath(url));
    await streamPipeline(res.body, dest);
    console.log(`✓ ${url}`);
  } catch (err) {
    console.warn(`✗ Error ${url}: ${err.message}`);
  }
}

async function runPool(tasks, limit) {
  const executing = [];
  for (const task of tasks) {
    const p = task();
    executing.push(p);
    if (executing.length >= limit) await Promise.race(executing);
    p.finally(() => {
      const idx = executing.indexOf(p);
      if (idx > -1) executing.splice(idx, 1);
    });
  }
  await Promise.all(executing);
}

await runPool(assets.map(u => () => download(u)), CONCURRENCY);

This implementation streams data directly to disk using pipeline to avoid memory exhaustion when handling large video files. The concurrency pool prevents overwhelming the target server while maximizing throughput.

Integration with the AI Workflow

The scripts/sync-skills.mjs utility propagates the download instruction from AGENTS.md to platform-specific AI agent configurations. During execution, the AI agent performs the following sequence:

1. Parses the target page's DOM to extract all src, srcset, and background-image URLs
2. Writes the asset manifest to a temporary JSON file
3. Invokes node scripts/download-assets.mjs to populate public/
4. References the local paths in generated component specs under src/components/

This ensures that components import assets using relative paths like import heroImg from '@/public/images/example.com/hero.jpg', eliminating external network dependencies during the build.

Key Files and Directories

File	Purpose
AGENTS.md	Central instruction set defining the "Download global assets" step (line 57)
README.md	High-level pipeline overview describing the asset download phase
scripts/download-assets.mjs	Node.js entry point for parallel asset acquisition (referenced in docs, generated at runtime)
.windsurf/workflows/clone-website.md	Workflow specification mentioning the download script (line 183)
scripts/sync-skills.mjs	Skill synchronization utility that propagates asset download instructions
public/images/	Destination for raster and vector images
public/videos/	Destination for video files
public/fonts/	Destination for font files

Summary

• The AI Website Cloner Template treats asset downloading as a first-class pipeline phase, distinct from component generation and orchestrated via AGENTS.md.
• The scripts/download-assets.mjs script implements concurrency-limited streaming downloads to efficiently acquire binary assets without memory bloat.
• Assets are automatically organized into semantic subdirectories (images/, videos/, fonts/) under public/ while preserving their original URL hierarchy.
• Generated components reference these local assets directly, ensuring the cloned site builds successfully without external CDN dependencies.

Frequently Asked Questions

How does the script handle assets that return 404 errors?

The download() function catches non-OK HTTP responses and connection errors, logging a warning (✗ Failed...) but continuing execution. This prevents missing assets from blocking the entire cloning process, allowing the pipeline to complete with partial asset coverage.

Can I modify the concurrency limit for slower networks?

Yes. The CONCURRENCY constant at the top of scripts/download-assets.mjs defaults to 4. Reduce this to 2 for unstable connections or increase it to 8 for high-bandwidth environments, though the default balances speed against rate-limiting concerns on target servers.

Where does the asset URL list come from?

The AI agent generates the list dynamically during the Reconnaissance phase by scanning the target page's DOM for img[src], video[src], source[srcset], and CSS background-image properties. This list is passed to the download script as a JSON array or hardcoded constant.

Is the download script included in the repository by default?

No. While the repository references scripts/download-assets.mjs in AGENTS.md and platform-specific workflows, the actual script is generated by the AI agent during the cloning process. You can customize the provided template code above and commit it to version control if you need reproducible manual runs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}