How Binary Assets Are Downloaded and Managed in AI Website Cloner

Binary assets are fetched using wget, stored in the public/ directory hierarchy, and verified by Docker health-checks before being served automatically by Next.js 16.

The JCodesMore/ai-website-cloner-template repository implements a streamlined pipeline for handling images, videos, and SEO assets. When cloning a website, the system must download and manage binary assets efficiently to ensure they are available for the Next.js application. This process leverages standard command-line tools and container health checks to guarantee reliability.

Asset Storage Structure

The repository maintains a strict folder hierarchy under public/ to organize binary assets by type. This structure ensures that the Next.js App Router can serve files without additional configuration.

The directory layout includes:

• public/images/ – Stores raster and vector images downloaded from the source site
• public/videos/ – Contains video files such as MP4 and WebM formats
• public/seo/ – Holds favicons, Open Graph images, and webmanifest files

To guarantee these folders exist in fresh clones, the repository includes .gitkeep placeholder files at public/images/.gitkeep, public/videos/.gitkeep, and public/seo/.gitkeep. This prevents Git from omitting empty directories during repository initialization.

Download Process with wget

When the clone-website command executes, it invokes a download routine that fetches each discovered binary URL using wget. The script writes responses directly to the appropriate subfolder inside public/.

The command format used by the implementation follows this pattern:

# Download an image to the public images folder

wget -P public/images https://example.com/assets/logo.png

# Download a video file

wget -P public/videos https://example.com/assets/intro.mp4

# Download SEO assets like favicons

wget -P public/seo https://example.com/favicon.ico

The -P flag directs wget to save files into the specified directory prefix, maintaining the asset organization required by the Next.js static file handler.

Docker Health-Check Verification

After downloading binary assets, the system verifies availability through Docker health-checks defined in docker-compose.yml. This ensures the Next.js dev server can serve all downloaded content before marking the container as healthy.

The health-check configuration uses wget to request the root endpoint:

# docker-compose.yml health-check configuration

healthcheck:
  test: ["CMD-SHELL", "wget -qO- http://localhost:3000/ || exit 1"]
  interval: 10s
  timeout: 5s
  retries: 3

This command requests http://localhost:3000/ and implicitly forces the server to resolve static asset URLs. If the request fails, the health-check exits with a non-zero status, triggering a container restart. This mechanism guarantees that all binary assets have been successfully stored and are reachable via the Next.js static file handler.

Serving Static Assets in Next.js

Once verified, Next.js 16 automatically maps any file under public/ to root URLs without requiring additional code. The framework handles binary asset delivery directly through its App Router.

For example, a file stored at public/images/logo.png becomes accessible at https://your-site.com/logo.png. Components reference these assets using direct paths:

// Next.js component referencing a downloaded image
import Image from "next/image";

export default function BrandLogo() {
  return (
    <Image
      src="/logo.png"               // Served from public/images/logo.png
      alt="Brand logo"
      width={200}
      height={60}
    />
  );
}

This zero-configuration approach leverages the standard Next.js public/ directory convention, eliminating the need for custom server routes or middleware to handle binary content.

Summary

• Binary assets are stored in public/images/, public/videos/, and public/seo/ with .gitkeep files ensuring directory persistence
• The clone-website command uses wget -P to download assets directly into their respective folders
• docker-compose.yml implements health-checks using wget -qO- http://localhost:3000/ to verify assets are servable
• Next.js 16 automatically serves files from public/ without additional configuration, mapping public/images/logo.png to /logo.png

Frequently Asked Questions

How does the clone-website command handle large video files?

The command uses standard wget without size restrictions, writing binary data directly to public/videos/. Since the operation happens during the build phase before container health checks, large files are downloaded completely before the Next.js server attempts to serve them.

What happens if a binary asset download fails during the cloning process?

If wget encounters a network error or invalid URL, it returns a non-zero exit code that propagates to the clone-website script. This prevents the workflow from completing successfully, and the Docker health-check will subsequently fail because the expected file will not exist in the public/ directory to be served.

Why are .gitkeep files used in the public folders?

The .gitkeep files at public/images/.gitkeep, public/videos/.gitkeep, and public/seo/.gitkeep ensure Git tracks these otherwise empty directories. This guarantees that when developers clone the repository, the expected folder structure already exists and the clone-website script can write downloaded assets to these locations without encountering "directory not found" errors.

Can binary assets be served from subdirectories within public/images/?

Yes, Next.js 16 supports nested directories within public/. The wget -P flag preserves relative paths when downloading, so an asset from https://example.com/assets/icons/icon.png downloaded to public/images/assets/icons/icon.png would be accessible at /assets/icons/icon.png in the browser.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}