# Best Practices for Writing Effective Prompts for Spider Creator: A Complete Guide

> Master Spider Creator prompts! Learn to write effective prompts with this guide. Generate accurate Playwright spiders by defining clear goals, sequential actions, and using code blocks.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: best-practices
- Published: 2026-02-26

---

**Start with a clear imperative goal, enumerate concrete actions in sequence, define sample sizes or stopping criteria, and wrap structured data in fenced code blocks to generate accurate Playwright-based web scraping spiders with Spider Creator.**

Spider Creator is an open-source framework that transforms natural-language task descriptions into fully-functional Playwright-based web-scraping spiders. Writing effective prompts for Spider Creator is crucial because each prompt serves as the sole source of truth for distinct stages of the LLM-driven pipeline, directly impacting the accuracy and maintainability of the generated spiders.

## Understanding the Spider Creator Prompt Pipeline

Spider Creator processes prompts through seven distinct stages, each defined in specific source files. Understanding these stages helps you write targeted prompts that align with each phase of the conversion workflow:

| Stage | Prompt Variable | Primary File | Purpose |
|-------|---------------|--------------|---------|
| **Browser-Use Task** | `TASK_PROMPT` | [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py) | Directs the Browser Use agent to navigate, click, and capture recordings. |
| **Spider Draft** | `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT` | [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) | Generates an initial Scrapy-style spider from recordings and Mermaid mind-maps. |
| **XPath Planning** | `XPATH_BUILDER_PLANNING_PROMPT` | [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) | Plans which XPaths to build, grouped by URL, producing a structured `Planning` model. |
| **Spider Combination** | `SPIDER_COMBINATION_PROMPT_INSTRUCTIONS` | [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) | Merges multiple draft spiders into a single, production-ready Playwright spider. |
| **Address Remapping** | `SPIDER_ADDRESS_REMAPPING_PROMPT` | [`pipeline/sp_addr_remapping.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_addr_remapping.py) | Rewrites URLs so the spider can run against locally-served HTML. |
| **Execution Verification** | `XPATH_EXECUTION_VERIFICATION_PROMPT` | [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) | Scores the spider's output against verification criteria. |

## 10 Essential Guidelines for Spider Creator Prompts

Follow these evidence-based guidelines derived from the Spider Creator source code to maximize spider accuracy and reduce hallucinations:

### 1. Start with a Concise, Imperative Description

LLMs parse the first few sentences as high-level intent. Begin with a direct command that establishes the goal.

```python

# Effective opening

"Navigate to {url} homepage and extract all product listings."

```

### 2. Enumerate Concrete Actions in Sequence

The Browser Use agent reproduces actions step-by-step. Missing steps cause incomplete recordings that propagate errors through [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py).

```python

# Ordered actions

"""
1. Extract all product cards visible on the page.
2. Click the "View details" link on the first three cards.
3. Capture additional attributes from the detail page.
"""

```

### 3. Use Placeholders for Runtime Injection

Keep prompts reusable by using `{}` placeholders and applying `str.format()` at runtime. This pattern appears throughout [`examples/product_listings_spider.py`](https://github.com/carlosplanchon/spidercreator/blob/main/examples/product_listings_spider.py).

```python
PRODUCT_LISTING_TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards...
""".format(url=url)

```

### 4. Define Sample Sizes and Stopping Criteria

Prevent infinite loops by specifying bounded scope. The `XPATH_BUILDER_PLANNING_PROMPT` expects explicit limits.

```python
"Select a representative sample of 3-5 products, then stop."

```

### 5. Wrap Structured Data in Fenced Code Blocks

Downstream prompts extract recordings, mind-maps, and code verbatim. Use explicit fences with language specifiers.

````markdown

```json
{recordings}

```

```mermaid
{mindmap}

```

````

### 6. Use Unambiguous, Domain-Specific Language

Ambiguity leads to incorrect selectors in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py). Specify element types explicitly.

```python

# Ambiguous

"Get the price"

# Precise

"Extract the text content of the price element within the product card div"

```

### 7. Declare Expected Output Formats

When prompts feed structured models like `Planning` or `Action` in [`xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/xpath_builder_planning.py), define field expectations explicitly.

```python
"""
For each action, provide:
- action_description: string describing the interaction
- example_xpaths_you_might_need: list of potential XPath selectors
- verify: boolean indicating if verification is needed
"""

```

### 8. Maintain Consistency Across Prompt Sections

The pipeline concatenates prompts (e.g., spider code plus user thoughts in [`sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/sp_combination.py)). Reuse established patterns to reduce friction.

### 9. Keep Prompts Under 1KB

Long prompts increase token usage and risk truncation. Summarize recordings and rely on generated mind-maps to convey detail efficiently.

### 10. Test Iteratively

Run small tasks first and inspect intermediate outputs from [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py). Refine wording based on missing fields before scaling up.

## Practical Example: Writing a Production-Ready Task Prompt

The following example from [`examples/product_listings_spider.py`](https://github.com/carlosplanchon/spidercreator/blob/main/examples/product_listings_spider.py) demonstrates all guidelines in practice:

```python
from main import create_spider

PRODUCT_LISTING_TASK_PROMPT = """
Navigate to {url} homepage.
Extract all product cards visible on the page, capturing:
- Product name
- Price (both before-discount and after-discount, if present)
- Brand
- Stock status
- Main image URL
- Link to the product detail page (if any)

Select a representative sample of 3-5 products:
  For each selected product:
  - Click its detail link (if available).
  - Extract any additional attributes (description, extra images, SKU).

Stop once you have gathered data for the sample set.
"""

url = "https://tiendainglesa.com.uy/"
browser_use_task = PRODUCT_LISTING_TASK_PROMPT.format(url=url)

create_spider(browser_use_task=browser_use_task)

```

**Key implementation details:**

*   **Imperative opening:** `Navigate to {url} homepage` establishes immediate intent.
*   **Structured data requirements:** Bullet points specify exact fields to extract, reducing ambiguity in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py).
*   **Bounded scope:** `3-5 products` and `Stop once you have gathered` prevent infinite loops during Browser Use execution.
*   **Runtime injection:** The `{url}` placeholder enables reuse across different domains.

## How Prompts Flow Through the Spider Creator Pipeline

Understanding the end-to-end flow helps you debug and optimize your prompts:

1.  **Task Definition:** You supply `browser_use_task` (the prompt string) via CLI or API.
2.  **Browser Recording:** [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py) passes your prompt to the Browser Use `Agent`, which executes the described actions and generates recordings.
3.  **Draft Generation:** [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) consumes the recordings and a generated Mermaid mind-map, using `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT` to produce initial Scrapy-style code.
4.  **XPath Planning:** [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) analyzes the draft and creates a structured `Planning` model specifying which XPaths to build for each URL.
5.  **Spider Combination:** [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) merges multiple drafts into a single Playwright spider using `SPIDER_COMBINATION_PROMPT_INSTRUCTIONS`, incorporating any user thoughts you provide.
6.  **Address Remapping:** (Optional) [`pipeline/sp_addr_remapping.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_addr_remapping.py) rewrites URLs for local testing environments.
7.  **Execution Verification:** [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) scores the final output against your original requirements using `XPATH_EXECUTION_VERIFICATION_PROMPT`.

Each stage treats your prompt as the authoritative specification, so precision at the input stage cascades through to the final spider quality.

## Summary

*   **Start with a concise, imperative goal** to establish clear intent for the Browser Use agent.
*   **Enumerate concrete actions in sequence** to ensure complete recordings that feed [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py).
*   **Use `{}` placeholders** for runtime values like URLs, applying `str.format()` for reusable prompt templates.
*   **Define explicit sample sizes and stopping criteria** to prevent infinite loops and control scope.
*   **Wrap structured data in fenced code blocks** with language specifiers to preserve formatting across pipeline stages.
*   **Keep prompts under 1KB** to avoid token truncation and reduce latency.
*   **Test iteratively** with small samples, inspecting intermediate outputs from [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) before scaling.

## Frequently Asked Questions

### What makes a Spider Creator prompt different from standard LLM prompts?

Spider Creator prompts serve as executable specifications for an autonomous agent pipeline rather than simple question-answer exchanges. Each prompt must account for downstream consumption by specific pipeline stages like [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) and [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py), requiring explicit structure, fenced code blocks for data interchange, and precise action sequencing that standard conversational prompts do not need.

### How do I handle dynamic content or JavaScript-heavy sites in my prompts?

When writing prompts for dynamic sites, explicitly instruct the Browser Use agent to wait for specific elements or states before extracting data. In [`record_activity.py`](https://github.com/carlosplanchon/spidercreator/blob/main/record_activity.py), the agent interprets instructions like "Wait for the product grid to load completely" or "Click the 'Load more' button until no new items appear" as explicit signals to handle JavaScript execution and AJAX calls, ensuring the recordings passed to [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) contain complete data.

### What should I do if the generated spider misses fields or extracts incorrect data?

First, inspect the intermediate output from [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) to verify that your original prompt's field requirements were preserved in the recordings. If fields are missing, refine your task prompt to use more specific, domain-specific language (e.g., "Extract the text content of the price element within the product card div" instead of "Get the price") and ensure you enumerate each required field as a bullet point to reduce ambiguity in the `XPATH_BUILDER_PLANNING_PROMPT` stage.

### How long should my task prompts be for optimal results?

Keep your primary task prompts under 1KB (approximately 500-800 tokens) to prevent truncation and minimize latency during the Browser Use execution phase. If your task requires extensive detail, summarize the recordings and rely on the generated Mermaid mind-map (processed by `DRAFT_SCRAPY_SPIDER_CREATION_PROMPT`) to convey complex site structure, while keeping the imperative instructions and field requirements concise and scannable.