Best Practices for Writing Effective Prompts for Spider Creator: A Complete Guide

Start with a clear imperative goal, enumerate concrete actions in sequence, define sample sizes or stopping criteria, and wrap structured data in fenced code blocks to generate accurate Playwright-based web scraping spiders with Spider Creator.

Spider Creator is an open-source framework that transforms natural-language task descriptions into fully-functional Playwright-based web-scraping spiders. Writing effective prompts for Spider Creator is crucial because each prompt serves as the sole source of truth for distinct stages of the LLM-driven pipeline, directly impacting the accuracy and maintainability of the generated spiders.

Understanding the Spider Creator Prompt Pipeline

Spider Creator processes prompts through seven distinct stages, each defined in specific source files. Understanding these stages helps you write targeted prompts that align with each phase of the conversion workflow:

Stage	Prompt Variable	Primary File	Purpose
Browser-Use Task	TASK_PROMPT	record_activity.py	Directs the Browser Use agent to navigate, click, and capture recordings.
Spider Draft	DRAFT_SCRAPY_SPIDER_CREATION_PROMPT	pipeline/spider_draft.py	Generates an initial Scrapy-style spider from recordings and Mermaid mind-maps.
XPath Planning	XPATH_BUILDER_PLANNING_PROMPT	pipeline/xpath_builder_planning.py	Plans which XPaths to build, grouped by URL, producing a structured Planning model.
Spider Combination	SPIDER_COMBINATION_PROMPT_INSTRUCTIONS	pipeline/sp_combination.py	Merges multiple draft spiders into a single, production-ready Playwright spider.
Address Remapping	SPIDER_ADDRESS_REMAPPING_PROMPT	pipeline/sp_addr_remapping.py	Rewrites URLs so the spider can run against locally-served HTML.
Execution Verification	XPATH_EXECUTION_VERIFICATION_PROMPT	pipeline/verify_sp_execution.py	Scores the spider's output against verification criteria.

10 Essential Guidelines for Spider Creator Prompts

Follow these evidence-based guidelines derived from the Spider Creator source code to maximize spider accuracy and reduce hallucinations:

1. Start with a Concise, Imperative Description

LLMs parse the first few sentences as high-level intent. Begin with a direct command that establishes the goal.

# Effective opening

"Navigate to {url} homepage and extract all product listings."

2. Enumerate Concrete Actions in Sequence

The Browser Use agent reproduces actions step-by-step. Missing steps cause incomplete recordings that propagate errors through pipeline/spider_draft.py.

# Ordered actions

"""
1. Extract all product cards visible on the page.
2. Click the "View details" link on the first three cards.
3. Capture additional attributes from the detail page.
"""

3. Use Placeholders for Runtime Injection

Keep prompts reusable by using {} placeholders and applying str.format() at runtime. This pattern appears throughout examples/product_listings_spider.py.

PRODUCT_LISTING_TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards...
""".format(url=url)

4. Define Sample Sizes and Stopping Criteria

Prevent infinite loops by specifying bounded scope. The XPATH_BUILDER_PLANNING_PROMPT expects explicit limits.

"Select a representative sample of 3-5 products, then stop."

5. Wrap Structured Data in Fenced Code Blocks

Downstream prompts extract recordings, mind-maps, and code verbatim. Use explicit fences with language specifiers.

```json
{recordings}

```

```mermaid
{mindmap}

```

6. Use Unambiguous, Domain-Specific Language

Ambiguity leads to incorrect selectors in pipeline/xpath_builder_planning.py. Specify element types explicitly.

# Ambiguous

"Get the price"

# Precise

"Extract the text content of the price element within the product card div"

7. Declare Expected Output Formats

When prompts feed structured models like Planning or Action in xpath_builder_planning.py, define field expectations explicitly.

"""
For each action, provide:
- action_description: string describing the interaction
- example_xpaths_you_might_need: list of potential XPath selectors
- verify: boolean indicating if verification is needed
"""

8. Maintain Consistency Across Prompt Sections

The pipeline concatenates prompts (e.g., spider code plus user thoughts in sp_combination.py). Reuse established patterns to reduce friction.

9. Keep Prompts Under 1KB

Long prompts increase token usage and risk truncation. Summarize recordings and rely on generated mind-maps to convey detail efficiently.

10. Test Iteratively

Run small tasks first and inspect intermediate outputs from pipeline/spider_draft.py. Refine wording based on missing fields before scaling up.

Practical Example: Writing a Production-Ready Task Prompt

The following example from examples/product_listings_spider.py demonstrates all guidelines in practice:

from main import create_spider

PRODUCT_LISTING_TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards visible on the page, capturing:
- Product name
- Price (both before-discount and after-discount, if present)
- Brand
- Stock status
- Main image URL
- Link to the product detail page (if any)

Select a representative sample of 3-5 products:
  For each selected product:
  - Click its detail link (if available).
  - Extract any additional attributes (description, extra images, SKU).

Stop once you have gathered data for the sample set.
"""

url = "https://tiendainglesa.com.uy/"
browser_use_task = PRODUCT_LISTING_TASK_PROMPT.format(url=url)

create_spider(browser_use_task=browser_use_task)

Key implementation details:

• Imperative opening: Navigate to {url} homepage establishes immediate intent.
• Structured data requirements: Bullet points specify exact fields to extract, reducing ambiguity in pipeline/xpath_builder_planning.py.
• Bounded scope: 3-5 products and Stop once you have gathered prevent infinite loops during Browser Use execution.
• Runtime injection: The {url} placeholder enables reuse across different domains.

How Prompts Flow Through the Spider Creator Pipeline

Understanding the end-to-end flow helps you debug and optimize your prompts:

1. Task Definition: You supply browser_use_task (the prompt string) via CLI or API.
2. Browser Recording: record_activity.py passes your prompt to the Browser Use Agent, which executes the described actions and generates recordings.
3. Draft Generation: pipeline/spider_draft.py consumes the recordings and a generated Mermaid mind-map, using DRAFT_SCRAPY_SPIDER_CREATION_PROMPT to produce initial Scrapy-style code.
4. XPath Planning: pipeline/xpath_builder_planning.py analyzes the draft and creates a structured Planning model specifying which XPaths to build for each URL.
5. Spider Combination: pipeline/sp_combination.py merges multiple drafts into a single Playwright spider using SPIDER_COMBINATION_PROMPT_INSTRUCTIONS, incorporating any user thoughts you provide.
6. Address Remapping: (Optional) pipeline/sp_addr_remapping.py rewrites URLs for local testing environments.
7. Execution Verification: pipeline/verify_sp_execution.py scores the final output against your original requirements using XPATH_EXECUTION_VERIFICATION_PROMPT.

Each stage treats your prompt as the authoritative specification, so precision at the input stage cascades through to the final spider quality.

Summary

• Start with a concise, imperative goal to establish clear intent for the Browser Use agent.
• Enumerate concrete actions in sequence to ensure complete recordings that feed pipeline/spider_draft.py.
• Use {} placeholders for runtime values like URLs, applying str.format() for reusable prompt templates.
• Define explicit sample sizes and stopping criteria to prevent infinite loops and control scope.
• Wrap structured data in fenced code blocks with language specifiers to preserve formatting across pipeline stages.
• Keep prompts under 1KB to avoid token truncation and reduce latency.
• Test iteratively with small samples, inspecting intermediate outputs from pipeline/spider_draft.py before scaling.

Frequently Asked Questions

What makes a Spider Creator prompt different from standard LLM prompts?

Spider Creator prompts serve as executable specifications for an autonomous agent pipeline rather than simple question-answer exchanges. Each prompt must account for downstream consumption by specific pipeline stages like pipeline/xpath_builder_planning.py and pipeline/sp_combination.py, requiring explicit structure, fenced code blocks for data interchange, and precise action sequencing that standard conversational prompts do not need.

How do I handle dynamic content or JavaScript-heavy sites in my prompts?

When writing prompts for dynamic sites, explicitly instruct the Browser Use agent to wait for specific elements or states before extracting data. In record_activity.py, the agent interprets instructions like "Wait for the product grid to load completely" or "Click the 'Load more' button until no new items appear" as explicit signals to handle JavaScript execution and AJAX calls, ensuring the recordings passed to pipeline/spider_draft.py contain complete data.

What should I do if the generated spider misses fields or extracts incorrect data?

First, inspect the intermediate output from pipeline/spider_draft.py to verify that your original prompt's field requirements were preserved in the recordings. If fields are missing, refine your task prompt to use more specific, domain-specific language (e.g., "Extract the text content of the price element within the product card div" instead of "Get the price") and ensure you enumerate each required field as a bullet point to reduce ambiguity in the XPATH_BUILDER_PLANNING_PROMPT stage.

How long should my task prompts be for optimal results?

Keep your primary task prompts under 1KB (approximately 500-800 tokens) to prevent truncation and minimize latency during the Browser Use execution phase. If your task requires extensive detail, summarize the recordings and rely on the generated Mermaid mind-map (processed by DRAFT_SCRAPY_SPIDER_CREATION_PROMPT) to convey complex site structure, while keeping the imperative instructions and field requirements concise and scannable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}