How Website Content is Fetched and Processed in the AI Website Cloner

The /clone-website skill orchestrates a multi-phase AI pipeline that uses headless browser automation to extract design tokens, download assets to public/ directories, and generate component specifications for pixel-perfect website replication.

The JCodesMore/ai-website-cloner-template repository provides a Next.js template where website content fetching and processing is handled entirely by AI coding agents rather than traditional web scrapers. When you invoke the /clone-website command, the system executes a sophisticated workflow defined in .claude/skills/clone-website/SKILL.md that captures visual data, archives media files, and reverse-engineers component styles. This analysis explains exactly how website content is fetched and processed through the repository's agent-driven architecture.

The Five-Phase Content Acquisition Pipeline

According to the repository's README.md (lines 86-95), the cloning workflow follows a strict multi-phase pipeline orchestrated by AI agents. Each phase handles specific aspects of content extraction and transformation.

Phase 1: Reconnaissance

The agent initiates content fetching by opening the target URL in a headless browser instance. During this phase, the system:

• Captures full-page screenshots and viewport-specific renders
• Extracts design tokens including color palettes, font families, and spacing scales
• Performs an interaction sweep to document scroll behaviors, click handlers, hover states, and responsive breakpoints

Phase 2: Foundation

During the Foundation phase, the agent handles bulk asset acquisition. The AI issues HTTP requests to download all discovered resources—including images, videos, favicons, and fonts—and organizes them into the project's static directories. Simultaneously, the agent updates the global style files to match the source website's CSS variables and typography configurations.

Phase 3: Component Specs

The agent generates detailed specification files under docs/research/components/ that serve as blueprints for reconstruction. These specifications contain:

• Exact getComputedStyle() values for every DOM element
• Interaction models documenting state transitions and animations
• Multi-state content representations (default, hover, active, disabled)
• Responsive breakpoint matrices defining layout shifts

Phase 4: Parallel Build

Individual builder agents receive component specifications and generate corresponding React/Next.js code. Each agent works in an isolated git worktree to prevent conflicts, translating the static design tokens and interaction models into functional JSX components with Tailwind CSS or styled-components.

Phase 5: Assembly & QA

In the final phase, the system merges all isolated worktrees into the main src/app/ directory, wires the components into page routes, and executes a visual diff comparison against the original website to verify pixel-perfect fidelity before finalizing the cloned output.

Asset Storage and Project Structure

Downloaded website content is organized into specific directories within the generated Next.js project structure:

• public/images/ – Stores all raster graphics, icons, and background images pulled from the source site
• public/videos/ – Contains MP4, WebM, and other video assets discovered during the Foundation phase
• public/seo/ – Houses favicons, Open Graph images, and other metadata-related files
• docs/research/components/ – Holds the JSON/YAML specification files generated during Phase 3

The src/app/ directory serves as the destination for the processed React components that consume these static assets.

Skill Configuration and Synchronization

The cloning logic itself is not hardcoded in Python or TypeScript source files. Instead, the behavior is defined in .claude/skills/clone-website/SKILL.md, which serves as the source-of-truth instruction set for the AI agent. This markdown file contains the high-level directives that guide the agent through the five-phase pipeline.

When modifications are made to the skill definition, the scripts/sync-skills.mjs script regenerates platform-specific copies and synchronizes the updated instructions across all supported AI coding platforms. This architecture ensures that content fetching rules remain consistent regardless of which AI agent executes the clone command.

Triggering the Clone Process

To initiate the content fetching and processing workflow, start your AI coding agent with browser capabilities enabled:

# Launch Claude Code with Chrome integration

claude --chrome

Once the session is active, invoke the skill with your target URL:

/clone-website https://example.com

The agent will automatically execute the full pipeline, from initial reconnaissance to final assembly, depositing all fetched assets and generated code into the appropriate repository directories.

Summary

• Agent-driven fetching: Content acquisition is performed by AI agents following instructions in .claude/skills/clone-website/SKILL.md, not by standalone scraper scripts
• Structured asset storage: Downloaded media organizes into public/images/, public/videos/, and public/seo/ during the Foundation phase
• Computed style extraction: The system captures exact getComputedStyle() values and stores them in specification files under docs/research/components/
• Cross-platform synchronization: Skill updates propagate via scripts/sync-skills.mjs to ensure consistent processing logic across environments

Frequently Asked Questions

Does the repository include a custom web scraper for fetching content?

No. The JCodesMore/ai-website-cloner-template does not contain a traditional web scraper written in Python or Node.js. Instead, the /clone-website skill defined in SKILL.md instructs AI coding agents to perform content fetching using headless browser automation and HTTP requests directly during the orchestrated pipeline phases.

Where are downloaded images and media files stored?

All binary assets downloaded during the Foundation phase are stored in the public/ directory of the generated Next.js project. Specifically, images reside in public/images/, video files in public/videos/, and SEO-related assets like favicons in public/seo/. These paths are automatically referenced by the generated React components in src/app/.

What specific data does the Reconnaissance phase extract from the target website?

During Reconnaissance, the AI agent captures screenshots, extracts design tokens (colors, font families, spacing values), and maps interaction behaviors including scroll positions, click targets, hover states, and responsive breakpoints. This data feeds into the component specifications that ensure visual fidelity in the final cloned output.

How is the cloning skill synchronized across different AI platforms?

The skill definition is synchronized using scripts/sync-skills.mjs, which regenerates platform-specific copies of the skill instructions after any edits to the master SKILL.md file. This ensures that the content fetching and processing logic remains consistent whether the skill runs on Claude Code, Cursor, or other supported AI coding environments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}