# spidercreator | Carlos A. Planchón | Knowledge Base | Instagit

Automated web scraping spider generation using Browser Use and LLMs. Streamline the creation of Playwright-based spiders with minimal manual coding. Ideal for large enterprises with recurring data extraction needs.

GitHub Stars: 213

Repository: https://github.com/carlosplanchon/spidercreator

---

## Articles

### [Best Practices for Writing Effective Prompts for Spider Creator: A Complete Guide](/carlosplanchon/spidercreator/best-practices-for-writing-effective-prompts-for-spider-creator)

Master Spider Creator prompts! Learn to write effective prompts with this guide. Generate accurate Playwright spiders by defining clear goals, sequential actions, and using code blocks.

- Tags: best-practices
- Published: 2026-02-26

### [How to Extend Spider Creator by Adding Custom Pipeline Stages](/carlosplanchon/spidercreator/how-to-extend-spider-creator-by-adding-custom-pipeline-stages)

Extend Spider Creator by adding custom pipeline stages. Learn how to create a new Python module, expose a callable, and register it to inject your own logic into the execution flow.

- Tags: how-to-guide
- Published: 2026-02-26

### [How to Configure Generated Spiders to Handle Rate Limiting and Captchas in SpiderCreator](/carlosplanchon/spidercreator/how-generated-spiders-can-be-configured-to-handle-rate-limiting-and-captchas)

Configure generated spiders in SpiderCreator to handle rate limiting and captchas by modifying LLM prompts or post-processing Scrapy output. Learn how to integrate custom settings and middleware.

- Tags: how-to-guide
- Published: 2026-02-26

### [Structured and Non-Structured Planning Approaches in Spider Creator: Key Differences Explained](/carlosplanchon/spidercreator/difference-between-structured-and-non-structured-planning-approaches-in-spider-creator)

Understand structured vs non-structured planning in Spider Creator. Discover how raw text or Pydantic models drive your automated pipeline. Learn the key differences now.

- Tags: deep-dive
- Published: 2026-02-26

### [How to Export and Run Generated Spider Code as a Standalone Script](/carlosplanchon/spidercreator/how-to-export-and-run-generated-spider-code-standalone)

Export and run generated spider code as a standalone script easily. Learn how to convert LLM code into a self-contained Python script using make_cand_spider_runnable.

- Tags: how-to-guide
- Published: 2026-02-26

### [How Spider Creator Handles Dynamic Content and Infinite Scroll Pages](/carlosplanchon/spidercreator/how-spider-creator-handles-dynamic-content-and-infinite-scroll-pages)

Spider Creator records browser interactions like scrolls and JS executions to generate Scrapy spiders that replicate dynamic content and infinite scroll page behavior using Playwright or Selenium.

- Tags: how-to-guide
- Published: 2026-02-26

### [Recommended DOM Chunk Size and ROI Thresholds in Spider Creator](/carlosplanchon/spidercreator/recommended-configurations-for-dom-chunk-size-and-roi-thresholds)

Discover optimal Spider Creator configurations: 32768 character DOM chunk size and 75 region ROI threshold balance memory efficiency and LLM costs for effective region detection.

- Tags: best-practices
- Published: 2026-02-26

### [How Mindmap Generation Works in Spider Creator: A Technical Deep Dive](/carlosplanchon/spidercreator/how-mindmap-generation-works-within-spider-creator-architecture)

Explore Spider Creator's technical deep dive on generative mindmaps. Understand how browser recordings, LangChain LLM, and Pydantic create structured visual Mermaid mindmaps.

- Tags: deep-dive
- Published: 2026-02-26

### [How to Test and Validate Generated Spiders in SpiderCreator: A 3-Stage Verification Pipeline](/carlosplanchon/spidercreator/how-to-effectively-test-and-validate-output-of-generated-spiders)

Learn how SpiderCreator validates Playwright spiders using a 3-stage pipeline: isolated execution, LLM scoring, and deterministic ranking for top quality.

- Tags: testing
- Published: 2026-02-26

### [How to Integrate Generated Spiders with the Scrapy Framework: A Complete Guide](/carlosplanchon/spidercreator/process-for-integrating-generated-spiders-with-scrapy-framework)

Learn to integrate generated spiders with Scrapy. Use SpiderCreator to record navigation, generate a Scrapy spider draft, and add it to your project for efficient web scraping.

- Tags: how-to-guide
- Published: 2026-02-26

### [How Spider Creator Uses Playwright to Scrape JavaScript-Rendered Pages](/carlosplanchon/spidercreator/how-spider-creators-playwright-integration-handles-javascript-rendered-pages)

Spider Creator harnesses Playwright's three-layer architecture to scrape JavaScript-rendered pages. Discover how it captures, processes, and extracts data from dynamic websites efficiently.

- Tags: deep-dive
- Published: 2026-02-26

### [How to Add Custom Extraction Rules to Spiders Generated by Spider Creator](/carlosplanchon/spidercreator/how-to-add-custom-extraction-rules-to-generated-spiders)

Learn how to add custom extraction rules to Spider Creator spiders. Modify Action objects and example xpaths for tailored web scraping before pipeline generation.

- Tags: how-to-guide
- Published: 2026-02-26

### [How to Tune Performance for Large-Scale Web Scraping Projects with SpiderCreator](/carlosplanchon/spidercreator/strategies-to-tune-performance-for-large-scale-web-scraping-projects)

Optimize large-scale web scraping performance with SpiderCreator. Learn strategies for DOM chunking ROI scope verification delays and parallel execution.

- Tags: performance
- Published: 2026-02-26

### [How to Optimize LLM Costs During Spider Generation in SpiderCreator](/carlosplanchon/spidercreator/how-to-optimize-llm-costs-during-spider-generation-process)

Optimize LLM costs in SpiderCreator with cheaper models, caching, prompt trimming, batching, and structured outputs. Reduce expenses during spider generation effectively.

- Tags: optimization
- Published: 2026-02-26

### [Spider Verification Failures in SpiderCreator: Common Causes and Fixes](/carlosplanchon/spidercreator/common-reasons-for-spider-verification-failures-and-how-to-fix-them)

Fix spider verification failures in SpiderCreator. Learn common causes like empty outputs or XPath issues and find solutions for normalizing output and validating plans.

- Tags: troubleshooting
- Published: 2026-02-26

### [How to Debug XPath Extraction Issues in Spiders Generated by Spider Creator](/carlosplanchon/spidercreator/how-to-debug-xpath-extraction-issues-in-spiders-generated-by-spider-creator)

Fix XPath extraction problems in Spider Creator. Inspect the plan, verify spider execution, and adjust DOM chunk sizes or prompts for accurate data results.

- Tags: how-to-guide
- Published: 2026-02-26

### [How to Configure the API Port and Set Custom Task IDs in Spider Creator](/carlosplanchon/spidercreator/how-to-configure-api-port-and-set-custom-task-ids-in-spider-creator)

Learn to configure the API port and set custom task IDs in Spider Creator. Guide covers adjusting port numbers for receive_bu_data.py and record_activity.py and assigning unique task IDs for organized data.

- Tags: how-to-guide
- Published: 2026-02-26

### [How to Implement Multi-Page Scraping in SpiderCreator: Listings + Details Guide](/carlosplanchon/spidercreator/how-to-implement-multi-page-scraping-within-single-spider)

Learn multi-page scraping in SpiderCreator. Automate listing and detail page extraction with generated Scrapy spiders. Analyze browser recordings for seamless data collection.

- Tags: how-to-guide
- Published: 2026-02-26

### [How to Customize the create_spider() Function for Advanced Use Cases in SpiderCreator](/carlosplanchon/spidercreator/how-to-customize-create_spider-function-for-advanced-use-cases)

Learn to customize the create_spider() function in SpiderCreator for advanced use cases. Extend RecordingAPIContext, use custom hooks, or modify pipeline stages for custom logic and LLMs.

- Tags: how-to-guide
- Published: 2026-02-26

### [How the Candidate Spider Execution and Verification Pipeline Works in SpiderCreator](/carlosplanchon/spidercreator/how-candidate-spider-execution-and-verification-pipeline-functions)

Discover how SpiderCreator's candidate spider execution and verification pipeline turns AI drafts into Parsel scripts, tests them locally, and uses an LLM for scoring to select the best candidate.

- Tags: internals
- Published: 2026-02-26

### [How Spider Creator Handles DOM Representation and ROI Classification: A Technical Deep Dive](/carlosplanchon/spidercreator/how-spider-creator-handles-dom-representation-and-roi-classification)

Explore how Spider Creator uses betterhtmlchunking and LLMs for DOM representation and ROI classification. Understand its technical approach to web scraping and planning intent.

- Tags: deep-dive
- Published: 2026-02-26

### [How Browser Use Records Browser Activity for Spider Generation in SpiderCreator](/carlosplanchon/spidercreator/role-of-browser-use-in-recording-browser-activity-for-spider-generation)

Discover how Browser Use records browser activity for spider generation in SpiderCreator. This layer executes, observes, and streams LLM interactions for automated spider synthesis.

- Tags: how-to-guide
- Published: 2026-02-26

### [Spider Creator Multi-Stage XPath Planning: How It Works Internally](/carlosplanchon/spidercreator/how-spider-creators-multi-stage-xpath-planning-works-internally)

Discover how Spider Creator's multi-stage XPath planning works internally. Learn about its two-phase LLM approach for reliable data extraction plans. Read now.

- Tags: internals
- Published: 2026-02-26