# How to Handle Multi-URL Cloning with Isolated Per-Site Research Folders in AI Website Cloner > Easily clone multiple websites using AI Website Cloner. Learn how isolated per-site research folders prevent asset collisions and enable parallel processing for efficient web scraping. - Repository: [JCodesMore/ai-website-cloner-template](https://github.com/JCodesMore/ai-website-cloner-template) - Tags: how-to-guide - Published: 2026-07-01 --- **The `/clone-website` skill automatically creates hostname-scoped folders under `docs/research/` and `docs/design-references/` for each target URL, enabling parallel processing while preventing asset collisions between sites.** In the `JCodesMore/ai-website-cloner-template` repository, the `/clone-website` skill supports **multi-URL cloning** through a single command. When you pass multiple URLs, the pipeline validates each target and provisions isolated research directories derived from each hostname, ensuring that screenshots, design tokens, and component specifications from different sites never overwrite each other. ## How Multi-URL Cloning Works ### URL Parsing and Validation The skill begins by reading the argument string and splitting it on whitespace to identify individual targets. Each entry is normalized to a full URL and validated for reachability using the **browser-MCP tool**. This pre-flight check occurs in step 2 of the workflow defined in [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) (Pre-Flight section). ### Hostname-Derived Folder Isolation For every valid URL, the skill automatically creates dedicated directories based on the hostname: ```text docs/research// docs/design-references// ``` This isolation strategy is defined in [`.github/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.github/skills/clone-website/SKILL.md) under Scope Defaults. By namespacing artifacts under their respective hostnames, the system guarantees that files like [`BEHAVIORS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/BEHAVIORS.md) or component specs from one site cannot collide with identically named files from another. ### Parallel Processing with Git Worktrees After establishing output folders, the skill runs each site in **parallel by default**. Parallelism is achieved by spawning a separate **git worktree** for each site’s builder agents, allowing concurrent execution while the extractor continues processing the next site. If resource constraints require sequential processing, pass the `--sequential` flag to force the pipeline to complete one site fully before starting the next. ### Per-Site Extraction Pipeline Each URL follows the complete five-phase pipeline independently: 1. Reconnaissance 2. Foundation 3. Component Specs 4. Parallel Build 5. Assembly & QA Because folders are namespaced by hostname, every phase writes to isolated locations: - `docs/research//BEHAVIORS.md` – interaction model for that specific site. - `docs/research//components/.spec.md` – component specifications stored under the hostname folder. - `public/images//…` – downloaded assets organized by source. ### Final Merge and Build Once all builder worktrees finish, the main branch merges each site’s worktree. The final Next.js build runs once, containing **all cloned sites** side-by-side, with each site’s pages living under their respective routes. ## Why Per-Site Isolation Matters - **No naming clashes** – Two sites may use identical component names (e.g., `Header`). Because specs and assets remain under separate host folders, both coexist without overwriting each other. - **Clear audit trail** – Each site’s research artifacts are kept together, making it trivial to review extraction quality or re-run a single site without affecting others. - **Scalable parallelism** – Builders for different sites never block each other; the pipeline utilizes available CPU cores, dramatically cutting overall cloning time. ## Command Examples for Multi-URL Workflows ### 1. Clone Multiple Sites in Parallel (Default) ```bash /clone-website https://example.com https://another-site.org ``` This creates `docs/research/example.com/` and `docs/research/another-site.org/` automatically, processing both sites simultaneously. ### 2. Force Sequential Processing ```bash /clone-website https://example.com https://another-site.org --sequential ``` When the `--sequential` flag is supplied, the skill runs the first URL, waits for its full pipeline to finish, then starts the second. Use this on low-resource machines to prevent memory contention. ### 3. Inspect Per-Site Folders After Extraction ```bash # List generated research folders ls docs/research/ #> example.com another-site.org ``` ```bash # View a specific component spec from the first site cat docs/research/example.com/components/HeroSection.spec.md ``` ### 4. Re-Run a Single Site ```bash /clone-website https://example.com --only example.com ``` The `--only` flag tells the skill to reuse existing assets and only re-process the specified hostname, useful for iterating on a single site without regenerating others. ## Key Implementation Files | File | Role | |------|------| | [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) | Canonical definition of the `/clone-website` skill, including multi-URL handling and folder layout. | | [`.github/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.github/skills/clone-website/SKILL.md) | Platform-agnostic skill source used to generate the workflow for AI agents. | | [`README.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/README.md) | Documents the CLI syntax for passing multiple URLs (`/clone-website [ …]`). | | `scripts/sync-skills.mjs` | Regenerates the platform-specific skill files after any change to the source configuration. | ## Summary - Pass multiple URLs to `/clone-website` separated by whitespace to trigger multi-site processing. - The skill automatically creates isolated folders under `docs/research//` and `docs/design-references//` to prevent file collisions. - Parallel execution via git worktrees is the default; use `--sequential` for resource-constrained environments. - Each site runs through the full five-phase extraction pipeline independently, with artifacts stored under their respective hostname directories. - Use `--only ` to re-process specific sites without regenerating the entire batch. ## Frequently Asked Questions ### How does the tool prevent file collisions when cloning multiple sites? The system derives a folder name from each URL's hostname and creates isolated directories under `docs/research/` and `docs/design-references/`. According to the source code in [`.github/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.github/skills/clone-website/SKILL.md), this Scope Defaults rule ensures that components, behaviors, and assets from different sites remain physically separated, even when sites use identical naming conventions. ### Can I process multiple URLs sequentially instead of in parallel? Yes. While the default mode spawns git worktrees to process sites in parallel, you can force sequential execution by appending the `--sequential` flag to your command. This runs the full extraction pipeline for the first URL before beginning the second, reducing CPU and memory usage on low-resource machines. ### Where are the extracted assets stored for each cloned website? Assets are organized under hostname-scoped paths: `docs/research//` contains research artifacts like [`BEHAVIORS.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/BEHAVIORS.md) and component specs, while `public/images//` stores downloaded images. This structure is defined in [`.windsurf/workflows/clone-website.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.windsurf/workflows/clone-website.md) and ensures clear separation between sites. ### How do I update the skill configuration after modifying the source files? Run the `scripts/sync-skills.mjs` script to regenerate the platform-specific `/clone-website` skill definitions. This script synchronizes changes from [`.github/skills/clone-website/SKILL.md`](https://github.com/JCodesMore/ai-website-cloner-template/blob/main/.github/skills/clone-website/SKILL.md) to the workflow files used by AI agents, ensuring your multi-URL handling logic remains consistent across the template.