# How Website Content is Fetched and Processed in the AI Website Cloner > Learn how AI website cloner fetches and processes website content. Discover detailed steps for design token extraction, asset downloads, and component specification generation for perfect replication. - Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template) - Tags: internals - Published: 2026-06-30 --- **The `/clone-website` skill orchestrates a multi-phase AI pipeline that uses headless browser automation to extract design tokens, download assets to `public/` directories, and generate component specifications for pixel-perfect website replication.** The `JCodesMore/ai-website-cloner-template` repository provides a Next.js template where website content fetching and processing is handled entirely by AI coding agents rather than traditional web scrapers. When you invoke the `/clone-website` command, the system executes a sophisticated workflow defined in [`.claude/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.claude/skills/clone-website/SKILL.md) that captures visual data, archives media files, and reverse-engineers component styles. This analysis explains exactly how website content is fetched and processed through the repository's agent-driven architecture. ## The Five-Phase Content Acquisition Pipeline According to the repository's [`README.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/README.md) (lines 86-95), the cloning workflow follows a strict multi-phase pipeline orchestrated by AI agents. Each phase handles specific aspects of content extraction and transformation. ### Phase 1: Reconnaissance The agent initiates content fetching by opening the target URL in a headless browser instance. During this phase, the system: - Captures full-page screenshots and viewport-specific renders - Extracts design tokens including color palettes, font families, and spacing scales - Performs an interaction sweep to document scroll behaviors, click handlers, hover states, and responsive breakpoints ### Phase 2: Foundation During the Foundation phase, the agent handles bulk asset acquisition. The AI issues HTTP requests to download all discovered resources—including images, videos, favicons, and fonts—and organizes them into the project's static directories. Simultaneously, the agent updates the global style files to match the source website's CSS variables and typography configurations. ### Phase 3: Component Specs The agent generates detailed specification files under `docs/research/components/` that serve as blueprints for reconstruction. These specifications contain: - Exact `getComputedStyle()` values for every DOM element - Interaction models documenting state transitions and animations - Multi-state content representations (default, hover, active, disabled) - Responsive breakpoint matrices defining layout shifts ### Phase 4: Parallel Build Individual builder agents receive component specifications and generate corresponding React/Next.js code. Each agent works in an isolated git worktree to prevent conflicts, translating the static design tokens and interaction models into functional JSX components with Tailwind CSS or styled-components. ### Phase 5: Assembly & QA In the final phase, the system merges all isolated worktrees into the main `src/app/` directory, wires the components into page routes, and executes a visual diff comparison against the original website to verify pixel-perfect fidelity before finalizing the cloned output. ## Asset Storage and Project Structure Downloaded website content is organized into specific directories within the generated Next.js project structure: - **`public/images/`** – Stores all raster graphics, icons, and background images pulled from the source site - **`public/videos/`** – Contains MP4, WebM, and other video assets discovered during the Foundation phase - **`public/seo/`** – Houses favicons, Open Graph images, and other metadata-related files - **`docs/research/components/`** – Holds the JSON/YAML specification files generated during Phase 3 The `src/app/` directory serves as the destination for the processed React components that consume these static assets. ## Skill Configuration and Synchronization The cloning logic itself is not hardcoded in Python or TypeScript source files. Instead, the behavior is defined in **[`.claude/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.claude/skills/clone-website/SKILL.md)**, which serves as the source-of-truth instruction set for the AI agent. This markdown file contains the high-level directives that guide the agent through the five-phase pipeline. When modifications are made to the skill definition, the **`scripts/sync-skills.mjs`** script regenerates platform-specific copies and synchronizes the updated instructions across all supported AI coding platforms. This architecture ensures that content fetching rules remain consistent regardless of which AI agent executes the clone command. ## Triggering the Clone Process To initiate the content fetching and processing workflow, start your AI coding agent with browser capabilities enabled: ```bash # Launch Claude Code with Chrome integration claude --chrome ``` Once the session is active, invoke the skill with your target URL: ```bash /clone-website https://example.com ``` The agent will automatically execute the full pipeline, from initial reconnaissance to final assembly, depositing all fetched assets and generated code into the appropriate repository directories. ## Summary - **Agent-driven fetching**: Content acquisition is performed by AI agents following instructions in [`.claude/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.claude/skills/clone-website/SKILL.md), not by standalone scraper scripts - **Structured asset storage**: Downloaded media organizes into `public/images/`, `public/videos/`, and `public/seo/` during the Foundation phase - **Computed style extraction**: The system captures exact `getComputedStyle()` values and stores them in specification files under `docs/research/components/` - **Cross-platform synchronization**: Skill updates propagate via `scripts/sync-skills.mjs` to ensure consistent processing logic across environments ## Frequently Asked Questions ### Does the repository include a custom web scraper for fetching content? No. The `JCodesMore/ai-website-cloner-template` does not contain a traditional web scraper written in Python or Node.js. Instead, the `/clone-website` skill defined in [`SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/SKILL.md) instructs AI coding agents to perform content fetching using headless browser automation and HTTP requests directly during the orchestrated pipeline phases. ### Where are downloaded images and media files stored? All binary assets downloaded during the Foundation phase are stored in the `public/` directory of the generated Next.js project. Specifically, images reside in `public/images/`, video files in `public/videos/`, and SEO-related assets like favicons in `public/seo/`. These paths are automatically referenced by the generated React components in `src/app/`. ### What specific data does the Reconnaissance phase extract from the target website? During Reconnaissance, the AI agent captures screenshots, extracts design tokens (colors, font families, spacing values), and maps interaction behaviors including scroll positions, click targets, hover states, and responsive breakpoints. This data feeds into the component specifications that ensure visual fidelity in the final cloned output. ### How is the cloning skill synchronized across different AI platforms? The skill definition is synchronized using **`scripts/sync-skills.mjs`**, which regenerates platform-specific copies of the skill instructions after any edits to the master [`SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/SKILL.md) file. This ensures that the content fetching and processing logic remains consistent whether the skill runs on Claude Code, Cursor, or other supported AI coding environments.