spidercreator

Master Spider Creator prompts! Learn to write effective prompts with this guide. Generate accurate Playwright spiders by defining clear goals, sequential actions, and using code blocks.

Extend Spider Creator by adding custom pipeline stages. Learn how to create a new Python module, expose a callable, and register it to inject your own logic into the execution flow.

Configure generated spiders in SpiderCreator to handle rate limiting and captchas by modifying LLM prompts or post-processing Scrapy output. Learn how to integrate custom settings and middleware.

Understand structured vs non-structured planning in Spider Creator. Discover how raw text or Pydantic models drive your automated pipeline. Learn the key differences now.

Export and run generated spider code as a standalone script easily. Learn how to convert LLM code into a self-contained Python script using make_cand_spider_runnable.

Spider Creator records browser interactions like scrolls and JS executions to generate Scrapy spiders that replicate dynamic content and infinite scroll page behavior using Playwright or Selenium.

Discover optimal Spider Creator configurations: 32768 character DOM chunk size and 75 region ROI threshold balance memory efficiency and LLM costs for effective region detection.

Explore Spider Creator's technical deep dive on generative mindmaps. Understand how browser recordings, LangChain LLM, and Pydantic create structured visual Mermaid mindmaps.

Learn how SpiderCreator validates Playwright spiders using a 3-stage pipeline: isolated execution, LLM scoring, and deterministic ranking for top quality.

Learn to integrate generated spiders with Scrapy. Use SpiderCreator to record navigation, generate a Scrapy spider draft, and add it to your project for efficient web scraping.

Spider Creator harnesses Playwright's three-layer architecture to scrape JavaScript-rendered pages. Discover how it captures, processes, and extracts data from dynamic websites efficiently.

Learn how to add custom extraction rules to Spider Creator spiders. Modify Action objects and example xpaths for tailored web scraping before pipeline generation.