# How to Add Custom Extraction Rules to Spiders Generated by Spider Creator

> Learn how to add custom extraction rules to Spider Creator spiders. Modify Action objects and example xpaths for tailored web scraping before pipeline generation.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can inject custom extraction rules into Spider Creator by modifying the `Action` objects in the structured planning stage, specifically by updating the `example_xpaths_you_might_need` field before the pipeline combines candidates into the final spider.**

Spider Creator is an open-source framework that automates Scrapy spider generation from browsing recordings. To add custom extraction rules to the spiders generated by Spider Creator, you need to intercept the structured planning data structure and inject your own XPath or CSS selectors before the candidate generation and combination stages execute.

## Understanding the Spider Creator Pipeline Architecture

The pipeline follows a multi-stage flow: recordings are converted to a mind-map, then to a draft spider, followed by **XPath-builder planning** where structured `Action` objects are created. These actions contain the `example_xpaths_you_might_need` list that dictates what data the LLM will extract. The pipeline then generates candidate spiders, verifies them, and combines the best candidates into the final script.

## Where to Inject Custom Extraction Rules in Spider Creator

The optimal injection point is immediately after the structured planning stage completes but before candidate generation begins. At this stage, the `Planning` object contains `InUrl` entries, each with an `action_list` of `Action` objects.

### The Action Model Structure

In [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py) (lines 47-56), the `Action` class defines the schema:

```python
class Action(BaseModel):
    action_description: str
    example_xpaths_you_might_need: list[str]   # Target field for custom rules

    verify: str                                # Verification hint

```

### Modifying the Action List

To add custom extraction rules, iterate through the planning structure and update the `example_xpaths_you_might_need` field:

```python
def inject_custom_xpaths(planning):
    for in_url in planning.in_url_list:
        for action in in_url.action_list:
            action.example_xpaths_you_might_need = [
                "//div[@class='price']/text()",
                "//h1[@class='title']/text()",
                "//img[@class='product-image']/@src"
            ]
            action.verify = "Check that price contains currency symbol"
    return planning

```

## Preserving Custom Rules During Spider Combination

By default, [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) (lines 34-48) contains `remove_example_xpaths_from_actions`, which strips the `example_xpaths_you_might_need` and `verify` fields before sending data to the combination LLM. To preserve your custom rules, modify this function or skip the removal step.

### Adapting the Removal Function

Update `remove_example_xpaths_from_actions` in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) to conditionally preserve custom rules:

```python
def remove_example_xpaths_from_actions(data):
    data_copy = copy.deepcopy(data)
    if "action_list" in data_copy:
        for action in data_copy["action_list"]:
            # Only remove auto-generated fields if no custom rules present

            if not action.get("custom_extraction_rules"):
                action.pop("example_xpaths_you_might_need", None)
                action.pop("verify", None)
    return data_copy

```

## Alternative Injection Points for Custom Extraction Rules

While the structured planning stage is the primary hook, you can also inject rules at earlier or later stages depending on your workflow.

### Draft Spider Stage

In [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) (lines 31-46), the `make_scrapy_spider_draft` function creates the initial spider template. You can patch the draft string directly before it proceeds to XPath planning, though this is less structured than modifying the `Action` objects.

### Candidate Generation Stage

In [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) (lines 19-34), the `classify_roi_html_create_cand_spider` function generates candidate spiders using the `SCRAPY_CREATION_PROMPT`. You can append custom extraction instructions to this prompt to influence how the LLM writes the spider code.

## Summary

- **Spider Creator** generates spiders through a structured pipeline: draft → planning → candidates → verification → combination.
- To add custom extraction rules, modify the **`example_xpaths_you_might_need`** field in `Action` objects after the XPath-builder planning stage.
- Prevent the default cleanup in **`remove_example_xpaths_from_actions`** (located in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py)) from stripping your custom rules.
- Alternative injection points include the draft spider stage ([`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)) and the candidate generation prompt ([`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)).

## Frequently Asked Questions

### How do I prevent Spider Creator from overwriting my custom XPaths?

The pipeline automatically removes `example_xpaths_you_might_need` in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) via the `remove_example_xpaths_from_actions` function. To preserve your custom XPaths, either comment out this function call in the main pipeline flow, or modify the function to skip removal when custom rules are detected.

### Can I use CSS selectors instead of XPath in Spider Creator?

While the `Action` model specifically includes `example_xpaths_you_might_need`, you can store CSS selectors in this list by prefixing them with `css:` or by adding a separate `custom_css_selectors` field to the `Action` model in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py). The LLM in the candidate generation stage will interpret these as extraction instructions.

### What is the best stage to inject custom extraction rules?

The most reliable injection point is immediately after the structured planning stage completes in [`pipeline/xpath_builder_planning.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/xpath_builder_planning.py). At this point, the `Planning` object contains fully structured `Action` objects with `example_xpaths_you_might_need` fields ready for modification, before the pipeline proceeds to candidate generation and combination.

### Does adding custom rules break the verification stage?

No, the verification stage in [`pipeline/verify_sp_execution.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verify_sp_execution.py) (lines 21-56) executes the generated spider and checks results against the `verify` field in the `Action` object. Since you can set custom verification hints when injecting your rules, the verification process will actually validate that your custom XPaths are extracting the correct data.