# How Binary Assets Are Downloaded and Managed in AI Website Cloner

> Learn how ai website cloner downloads and manages binary assets using wget and Docker. Discover efficient asset handling for your cloned websites.

- Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template)
- Tags: internals
- Published: 2026-07-11

---

**Binary assets are fetched using `wget`, stored in the `public/` directory hierarchy, and verified by Docker health-checks before being served automatically by Next.js 16.**

The `JCodesMore/ai-website-cloner-template` repository implements a streamlined pipeline for handling images, videos, and SEO assets. When cloning a website, the system must download and manage binary assets efficiently to ensure they are available for the Next.js application. This process leverages standard command-line tools and container health checks to guarantee reliability.

## Asset Storage Structure

The repository maintains a strict folder hierarchy under `public/` to organize binary assets by type. This structure ensures that the Next.js App Router can serve files without additional configuration.

The directory layout includes:
- **`public/images/`** – Stores raster and vector images downloaded from the source site
- **`public/videos/`** – Contains video files such as MP4 and WebM formats
- **`public/seo/`** – Holds favicons, Open Graph images, and `webmanifest` files

To guarantee these folders exist in fresh clones, the repository includes **`.gitkeep`** placeholder files at `public/images/.gitkeep`, `public/videos/.gitkeep`, and `public/seo/.gitkeep`. This prevents Git from omitting empty directories during repository initialization.

## Download Process with wget

When the **`clone-website`** command executes, it invokes a download routine that fetches each discovered binary URL using **`wget`**. The script writes responses directly to the appropriate subfolder inside `public/`.

The command format used by the implementation follows this pattern:

```bash

# Download an image to the public images folder

wget -P public/images https://example.com/assets/logo.png

# Download a video file

wget -P public/videos https://example.com/assets/intro.mp4

# Download SEO assets like favicons

wget -P public/seo https://example.com/favicon.ico

```

The `-P` flag directs `wget` to save files into the specified directory prefix, maintaining the asset organization required by the Next.js static file handler.

## Docker Health-Check Verification

After downloading binary assets, the system verifies availability through **Docker health-checks** defined in [`docker-compose.yml`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/docker-compose.yml). This ensures the Next.js dev server can serve all downloaded content before marking the container as healthy.

The health-check configuration uses `wget` to request the root endpoint:

```yaml

# docker-compose.yml health-check configuration

healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:3000/ || exit 1"]
  interval: 10s
  timeout: 5s
  retries: 3

```

This command requests `http://localhost:3000/` and implicitly forces the server to resolve static asset URLs. If the request fails, the health-check exits with a non-zero status, triggering a container restart. This mechanism guarantees that all binary assets have been successfully stored and are reachable via the Next.js static file handler.

## Serving Static Assets in Next.js

Once verified, **Next.js 16** automatically maps any file under `public/` to root URLs without requiring additional code. The framework handles binary asset delivery directly through its App Router.

For example, a file stored at `public/images/logo.png` becomes accessible at `https://your-site.com/logo.png`. Components reference these assets using direct paths:

```tsx
// Next.js component referencing a downloaded image
import Image from "next/image";

export default function BrandLogo() {
  return (
    <Image
      src="/logo.png"               // Served from public/images/logo.png
      alt="Brand logo"
      width={200}
      height={60}
    />
  );
}

```

This zero-configuration approach leverages the standard Next.js `public/` directory convention, eliminating the need for custom server routes or middleware to handle binary content.

## Summary

- **Binary assets** are stored in `public/images/`, `public/videos/`, and `public/seo/` with `.gitkeep` files ensuring directory persistence
- The **`clone-website`** command uses `wget -P` to download assets directly into their respective folders
- **[`docker-compose.yml`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/docker-compose.yml)** implements health-checks using `wget -qO- http://localhost:3000/` to verify assets are servable
- **Next.js 16** automatically serves files from `public/` without additional configuration, mapping `public/images/logo.png` to `/logo.png`

## Frequently Asked Questions

### How does the clone-website command handle large video files?

The command uses standard `wget` without size restrictions, writing binary data directly to `public/videos/`. Since the operation happens during the build phase before container health checks, large files are downloaded completely before the Next.js server attempts to serve them.

### What happens if a binary asset download fails during the cloning process?

If `wget` encounters a network error or invalid URL, it returns a non-zero exit code that propagates to the `clone-website` script. This prevents the workflow from completing successfully, and the Docker health-check will subsequently fail because the expected file will not exist in the `public/` directory to be served.

### Why are .gitkeep files used in the public folders?

The `.gitkeep` files at `public/images/.gitkeep`, `public/videos/.gitkeep`, and `public/seo/.gitkeep` ensure Git tracks these otherwise empty directories. This guarantees that when developers clone the repository, the expected folder structure already exists and the `clone-website` script can write downloaded assets to these locations without encountering "directory not found" errors.

### Can binary assets be served from subdirectories within public/images/?

Yes, Next.js 16 supports nested directories within `public/`. The `wget -P` flag preserves relative paths when downloading, so an asset from `https://example.com/assets/icons/icon.png` downloaded to `public/images/assets/icons/icon.png` would be accessible at `/assets/icons/icon.png` in the browser.