How to Add Custom Extraction Rules to Spiders Generated by Spider Creator

You can inject custom extraction rules into Spider Creator by modifying the Action objects in the structured planning stage, specifically by updating the example_xpaths_you_might_need field before the pipeline combines candidates into the final spider.

Spider Creator is an open-source framework that automates Scrapy spider generation from browsing recordings. To add custom extraction rules to the spiders generated by Spider Creator, you need to intercept the structured planning data structure and inject your own XPath or CSS selectors before the candidate generation and combination stages execute.

Understanding the Spider Creator Pipeline Architecture

The pipeline follows a multi-stage flow: recordings are converted to a mind-map, then to a draft spider, followed by XPath-builder planning where structured Action objects are created. These actions contain the example_xpaths_you_might_need list that dictates what data the LLM will extract. The pipeline then generates candidate spiders, verifies them, and combines the best candidates into the final script.

Where to Inject Custom Extraction Rules in Spider Creator

The optimal injection point is immediately after the structured planning stage completes but before candidate generation begins. At this stage, the Planning object contains InUrl entries, each with an action_list of Action objects.

The Action Model Structure

In pipeline/xpath_builder_planning.py (lines 47-56), the Action class defines the schema:

class Action(BaseModel):
    action_description: str
    example_xpaths_you_might_need: list[str]   # Target field for custom rules

    verify: str                                # Verification hint

Modifying the Action List

To add custom extraction rules, iterate through the planning structure and update the example_xpaths_you_might_need field:

def inject_custom_xpaths(planning):
    for in_url in planning.in_url_list:
        for action in in_url.action_list:
            action.example_xpaths_you_might_need = [
                "//div[@class='price']/text()",
                "//h1[@class='title']/text()",
                "//img[@class='product-image']/@src"
            ]
            action.verify = "Check that price contains currency symbol"
    return planning

Preserving Custom Rules During Spider Combination

By default, pipeline/sp_combination.py (lines 34-48) contains remove_example_xpaths_from_actions, which strips the example_xpaths_you_might_need and verify fields before sending data to the combination LLM. To preserve your custom rules, modify this function or skip the removal step.

Adapting the Removal Function

Update remove_example_xpaths_from_actions in pipeline/sp_combination.py to conditionally preserve custom rules:

def remove_example_xpaths_from_actions(data):
    data_copy = copy.deepcopy(data)
    if "action_list" in data_copy:
        for action in data_copy["action_list"]:
            # Only remove auto-generated fields if no custom rules present

            if not action.get("custom_extraction_rules"):
                action.pop("example_xpaths_you_might_need", None)
                action.pop("verify", None)
    return data_copy

Alternative Injection Points for Custom Extraction Rules

While the structured planning stage is the primary hook, you can also inject rules at earlier or later stages depending on your workflow.

Draft Spider Stage

In pipeline/spider_draft.py (lines 31-46), the make_scrapy_spider_draft function creates the initial spider template. You can patch the draft string directly before it proceeds to XPath planning, though this is less structured than modifying the Action objects.

Candidate Generation Stage

In pipeline/roiclf_spcandmkr.py (lines 19-34), the classify_roi_html_create_cand_spider function generates candidate spiders using the SCRAPY_CREATION_PROMPT. You can append custom extraction instructions to this prompt to influence how the LLM writes the spider code.

Summary

• Spider Creator generates spiders through a structured pipeline: draft → planning → candidates → verification → combination.
• To add custom extraction rules, modify the example_xpaths_you_might_need field in Action objects after the XPath-builder planning stage.
• Prevent the default cleanup in remove_example_xpaths_from_actions (located in pipeline/sp_combination.py) from stripping your custom rules.
• Alternative injection points include the draft spider stage (pipeline/spider_draft.py) and the candidate generation prompt (pipeline/roiclf_spcandmkr.py).

Frequently Asked Questions

How do I prevent Spider Creator from overwriting my custom XPaths?

The pipeline automatically removes example_xpaths_you_might_need in pipeline/sp_combination.py via the remove_example_xpaths_from_actions function. To preserve your custom XPaths, either comment out this function call in the main pipeline flow, or modify the function to skip removal when custom rules are detected.

Can I use CSS selectors instead of XPath in Spider Creator?

While the Action model specifically includes example_xpaths_you_might_need, you can store CSS selectors in this list by prefixing them with css: or by adding a separate custom_css_selectors field to the Action model in pipeline/xpath_builder_planning.py. The LLM in the candidate generation stage will interpret these as extraction instructions.

What is the best stage to inject custom extraction rules?

The most reliable injection point is immediately after the structured planning stage completes in pipeline/xpath_builder_planning.py. At this point, the Planning object contains fully structured Action objects with example_xpaths_you_might_need fields ready for modification, before the pipeline proceeds to candidate generation and combination.

Does adding custom rules break the verification stage?

No, the verification stage in pipeline/verify_sp_execution.py (lines 21-56) executes the generated spider and checks results against the verify field in the Action object. Since you can set custom verification hints when injecting your rules, the verification process will actually validate that your custom XPaths are extracting the correct data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}