# How to Automate Downloading and Organizing Assets from Target Websites with the AI Website Cloner Template

> Automate downloading and organizing website assets like images, videos, and fonts with the AI Website Cloner Template. Get a structured public directory for easy rebuilds.

- Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template)
- Tags: tutorial
- Published: 2026-06-28

---

**The AI Website Cloner Template automates asset acquisition by generating a Node.js script (`scripts/download-assets.mjs`) that parallel-downloads images, videos, and fonts from the target site into a structured `public/` directory, enabling pixel-perfect Next.js rebuilds.**

The **AI Website Cloner Template** converts any live website into a fully functional Next.js application. A critical phase in this pipeline is the automated extraction and organization of binary assets—images, videos, SVGs, and fonts—from the target URL into the project's `public/` folder. This process ensures that the cloned site maintains visual fidelity without relying on external CDN links.

## Asset Download Automation Architecture

### Pipeline Phase Integration

According to [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md) (line 57), the cloning workflow includes a mandatory "Download global assets" step that occurs after visual token extraction but before component generation. This step writes and executes `scripts/download-assets.mjs` to fetch all external resources enumerated by the AI agent during the **Reconnaissance** phase.

### Directory Structure and Routing

The template enforces strict file-type routing under `public/` to mirror the source site’s asset organization:

- **Images** (PNG, JPG, SVG, WEBP, GIF) → `public/images/`
- **Videos** (MP4, WEBM, OGG) → `public/videos/`
- **Fonts** (WOFF, WOFF2, TTF, OTF) → `public/fonts/`
- **Miscellaneous** binaries → `public/misc/`

Each asset preserves its remote directory structure under its type folder, mapping `https://example.com/assets/img/logo.png` to `public/images/example.com/assets/img/logo.png`.

## Implementing the Asset Download Script

While the repository references `scripts/download-assets.mjs` in documentation—notably at [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) (line 183) and [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md) (line 57)—the file is generated dynamically during the cloning process. The script implements four core design principles that ensure efficient and reliable asset acquisition:

1. **Batch parallelism** – Concurrent downloads limited to 4 connections to prevent network saturation
2. **Error resilience** – Non-blocking failure handling with warning logs for 404s or timeouts
3. **Path preservation** – Replication of remote URL paths under the local `public/` root
4. **Type-based routing** – Automatic categorization into `images/`, `videos/`, or `fonts/` subdirectories

### Complete Script Implementation

```javascript
// scripts/download-assets.mjs
import { createWriteStream, mkdirSync } from 'node:fs';
import { pipeline } from 'node:stream';
import { promisify } from 'node:util';
import { URL } from 'node:url';
import fetch from 'node-fetch';

const streamPipeline = promisify(pipeline);
const CONCURRENCY = 4;

// Asset list generated by the AI agent from DOM analysis
const assets = [
  'https://example.com/images/hero.jpg',
  'https://example.com/videos/intro.mp4',
  'https://example.com/fonts/inter.woff2'
];

function getLocalPath(urlStr) {
  const url = new URL(urlStr);
  const pathname = url.pathname.replace(/^\/+/, '');
  const ext = pathname.split('.').pop().toLowerCase();

  let baseDir = 'public';
  if (['png', 'jpg', 'jpeg', 'webp', 'svg', 'gif'].includes(ext)) baseDir += '/images';
  else if (['mp4', 'webm', 'ogg'].includes(ext)) baseDir += '/videos';
  else if (['woff', 'woff2', 'ttf', 'otf'].includes(ext)) baseDir += '/fonts';
  else baseDir += '/misc';

  const fullPath = `${baseDir}/${url.host}/${pathname}`;
  mkdirSync(fullPath.substring(0, fullPath.lastIndexOf('/')), { recursive: true });
  return fullPath;
}

async function download(url) {
  try {
    const res = await fetch(url);
    if (!res.ok) {
      console.warn(`✗ Failed ${url}: ${res.status}`);
      return;
    }
    const dest = createWriteStream(getLocalPath(url));
    await streamPipeline(res.body, dest);
    console.log(`✓ ${url}`);
  } catch (err) {
    console.warn(`✗ Error ${url}: ${err.message}`);
  }
}

async function runPool(tasks, limit) {
  const executing = [];
  for (const task of tasks) {
    const p = task();
    executing.push(p);
    if (executing.length >= limit) await Promise.race(executing);
    p.finally(() => {
      const idx = executing.indexOf(p);
      if (idx > -1) executing.splice(idx, 1);
    });
  }
  await Promise.all(executing);
}

await runPool(assets.map(u => () => download(u)), CONCURRENCY);

```

This implementation streams data directly to disk using `pipeline` to avoid memory exhaustion when handling large video files. The concurrency pool prevents overwhelming the target server while maximizing throughput.

## Integration with the AI Workflow

The `scripts/sync-skills.mjs` utility propagates the download instruction from [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md) to platform-specific AI agent configurations. During execution, the AI agent performs the following sequence:

1. Parses the target page's DOM to extract all `src`, `srcset`, and `background-image` URLs
2. Writes the asset manifest to a temporary JSON file
3. Invokes `node scripts/download-assets.mjs` to populate `public/`
4. References the local paths in generated component specs under `src/components/`

This ensures that components import assets using relative paths like `import heroImg from '@/public/images/example.com/hero.jpg'`, eliminating external network dependencies during the build.

## Key Files and Directories

| File | Purpose |
|------|---------|
| [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md) | Central instruction set defining the "Download global assets" step (line 57) |
| [`README.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/README.md) | High-level pipeline overview describing the asset download phase |
| `scripts/download-assets.mjs` | Node.js entry point for parallel asset acquisition (referenced in docs, generated at runtime) |
| [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) | Workflow specification mentioning the download script (line 183) |
| `scripts/sync-skills.mjs` | Skill synchronization utility that propagates asset download instructions |
| `public/images/` | Destination for raster and vector images |
| `public/videos/` | Destination for video files |
| `public/fonts/` | Destination for font files |

## Summary

- The AI Website Cloner Template treats asset downloading as a first-class pipeline phase, distinct from component generation and orchestrated via [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md).
- The `scripts/download-assets.mjs` script implements concurrency-limited streaming downloads to efficiently acquire binary assets without memory bloat.
- Assets are automatically organized into semantic subdirectories (`images/`, `videos/`, `fonts/`) under `public/` while preserving their original URL hierarchy.
- Generated components reference these local assets directly, ensuring the cloned site builds successfully without external CDN dependencies.

## Frequently Asked Questions

### How does the script handle assets that return 404 errors?

The `download()` function catches non-OK HTTP responses and connection errors, logging a warning (`✗ Failed...`) but continuing execution. This prevents missing assets from blocking the entire cloning process, allowing the pipeline to complete with partial asset coverage.

### Can I modify the concurrency limit for slower networks?

Yes. The `CONCURRENCY` constant at the top of `scripts/download-assets.mjs` defaults to 4. Reduce this to 2 for unstable connections or increase it to 8 for high-bandwidth environments, though the default balances speed against rate-limiting concerns on target servers.

### Where does the asset URL list come from?

The AI agent generates the list dynamically during the **Reconnaissance** phase by scanning the target page's DOM for `img[src]`, `video[src]`, `source[srcset]`, and CSS `background-image` properties. This list is passed to the download script as a JSON array or hardcoded constant.

### Is the download script included in the repository by default?

No. While the repository references `scripts/download-assets.mjs` in [`AGENTS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/AGENTS.md) and platform-specific workflows, the actual script is generated by the AI agent during the cloning process. You can customize the provided template code above and commit it to version control if you need reproducible manual runs.