How to Handle Multi-URL Cloning with Isolated Per-Site Research Folders in AI Website Cloner

The /clone-website skill automatically creates hostname-scoped folders under docs/research/ and docs/design-references/ for each target URL, enabling parallel processing while preventing asset collisions between sites.

In the JCodesMore/ai-website-cloner-template repository, the /clone-website skill supports multi-URL cloning through a single command. When you pass multiple URLs, the pipeline validates each target and provisions isolated research directories derived from each hostname, ensuring that screenshots, design tokens, and component specifications from different sites never overwrite each other.

How Multi-URL Cloning Works

URL Parsing and Validation

The skill begins by reading the argument string and splitting it on whitespace to identify individual targets. Each entry is normalized to a full URL and validated for reachability using the browser-MCP tool. This pre-flight check occurs in step 2 of the workflow defined in .windsurf/workflows/clone-website.md (Pre-Flight section).

Hostname-Derived Folder Isolation

For every valid URL, the skill automatically creates dedicated directories based on the hostname:

docs/research/<hostname>/
docs/design-references/<hostname>/

This isolation strategy is defined in .github/skills/clone-website/SKILL.md under Scope Defaults. By namespacing artifacts under their respective hostnames, the system guarantees that files like BEHAVIORS.md or component specs from one site cannot collide with identically named files from another.

Parallel Processing with Git Worktrees

After establishing output folders, the skill runs each site in parallel by default. Parallelism is achieved by spawning a separate git worktree for each site’s builder agents, allowing concurrent execution while the extractor continues processing the next site. If resource constraints require sequential processing, pass the --sequential flag to force the pipeline to complete one site fully before starting the next.

Per-Site Extraction Pipeline

Each URL follows the complete five-phase pipeline independently:

1. Reconnaissance
2. Foundation
3. Component Specs
4. Parallel Build
5. Assembly & QA

Because folders are namespaced by hostname, every phase writes to isolated locations:

• docs/research/<hostname>/BEHAVIORS.md – interaction model for that specific site.
• docs/research/<hostname>/components/<component>.spec.md – component specifications stored under the hostname folder.
• public/images/<hostname>/… – downloaded assets organized by source.

Final Merge and Build

Once all builder worktrees finish, the main branch merges each site’s worktree. The final Next.js build runs once, containing all cloned sites side-by-side, with each site’s pages living under their respective routes.

Why Per-Site Isolation Matters

• No naming clashes – Two sites may use identical component names (e.g., Header). Because specs and assets remain under separate host folders, both coexist without overwriting each other.
• Clear audit trail – Each site’s research artifacts are kept together, making it trivial to review extraction quality or re-run a single site without affecting others.
• Scalable parallelism – Builders for different sites never block each other; the pipeline utilizes available CPU cores, dramatically cutting overall cloning time.

Command Examples for Multi-URL Workflows

1. Clone Multiple Sites in Parallel (Default)

/clone-website https://example.com https://another-site.org

This creates docs/research/example.com/ and docs/research/another-site.org/ automatically, processing both sites simultaneously.

2. Force Sequential Processing

/clone-website https://example.com https://another-site.org --sequential

When the --sequential flag is supplied, the skill runs the first URL, waits for its full pipeline to finish, then starts the second. Use this on low-resource machines to prevent memory contention.

3. Inspect Per-Site Folders After Extraction

# List generated research folders

ls docs/research/
#> example.com  another-site.org

# View a specific component spec from the first site

cat docs/research/example.com/components/HeroSection.spec.md

4. Re-Run a Single Site

/clone-website https://example.com --only example.com

The --only flag tells the skill to reuse existing assets and only re-process the specified hostname, useful for iterating on a single site without regenerating others.

Key Implementation Files

File	Role
.windsurf/workflows/clone-website.md	Canonical definition of the /clone-website skill, including multi-URL handling and folder layout.
.github/skills/clone-website/SKILL.md	Platform-agnostic skill source used to generate the workflow for AI agents.
README.md	Documents the CLI syntax for passing multiple URLs (/clone-website <url1> [<url2> …]).
scripts/sync-skills.mjs	Regenerates the platform-specific skill files after any change to the source configuration.

Summary

• Pass multiple URLs to /clone-website separated by whitespace to trigger multi-site processing.
• The skill automatically creates isolated folders under docs/research/<hostname>/ and docs/design-references/<hostname>/ to prevent file collisions.
• Parallel execution via git worktrees is the default; use --sequential for resource-constrained environments.
• Each site runs through the full five-phase extraction pipeline independently, with artifacts stored under their respective hostname directories.
• Use --only <hostname> to re-process specific sites without regenerating the entire batch.

Frequently Asked Questions

How does the tool prevent file collisions when cloning multiple sites?

The system derives a folder name from each URL's hostname and creates isolated directories under docs/research/ and docs/design-references/. According to the source code in .github/skills/clone-website/SKILL.md, this Scope Defaults rule ensures that components, behaviors, and assets from different sites remain physically separated, even when sites use identical naming conventions.

Can I process multiple URLs sequentially instead of in parallel?

Yes. While the default mode spawns git worktrees to process sites in parallel, you can force sequential execution by appending the --sequential flag to your command. This runs the full extraction pipeline for the first URL before beginning the second, reducing CPU and memory usage on low-resource machines.

Where are the extracted assets stored for each cloned website?

Assets are organized under hostname-scoped paths: docs/research/<hostname>/ contains research artifacts like BEHAVIORS.md and component specs, while public/images/<hostname>/ stores downloaded images. This structure is defined in .windsurf/workflows/clone-website.md and ensures clear separation between sites.

How do I update the skill configuration after modifying the source files?

Run the scripts/sync-skills.mjs script to regenerate the platform-specific /clone-website skill definitions. This script synchronizes changes from .github/skills/clone-website/SKILL.md to the workflow files used by AI agents, ensuring your multi-URL handling logic remains consistent across the template.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}