# How Spider Creator Handles DOM Representation and ROI Classification: A Technical Deep Dive

> Explore how Spider Creator uses betterhtmlchunking and LLMs for DOM representation and ROI classification. Understand its technical approach to web scraping and planning intent.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: deep-dive
- Published: 2026-02-26

---

**Spider Creator converts raw HTML into a navigable DOM tree using the betterhtmlchunking library, then employs an LLM with structured output schemas to classify each Region of Interest (ROI) against user planning intent.**

The **carlosplanchon/spidercreator** repository implements a two-stage pipeline for intelligent web scraping. This article examines how the system builds a structured **DOM representation** from raw HTML and subsequently performs **ROI classification** to identify relevant page fragments for automatic spider generation.

## Building the DOM Representation with betterhtmlchunking

Spider Creator constructs a `DomRepresentation` object by processing raw HTML through the **betterhtmlchunking** library. The entry point is the `make_dom_representation` function in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py), which accepts a configurable `MAX_NODE_REPR_LENGTH` parameter to handle large documents.

The function wraps the HTML and initializes internal chunking via `dom_repr.start()`. The resulting object exposes two critical subsystems: the `tree_regions_system`, which discovers and sorts ROIs by their XPath position, and the `render_system`, which converts ROIs into consumable formats.

```python
from pipeline.make_dom_repr import make_dom_representation

dom_repr = make_dom_representation(
    website_html=website_html,
    MAX_NODE_REPR_LENGTH=32768,  # Configurable limit for node representation size

)

# Access sorted ROIs by XPath position

roi_count = len(dom_repr.tree_regions_system.sorted_roi_by_pos_xpath)
print(f"Discovered {roi_count} potential regions of interest")

```

The `render_system` provides two key methods: `get_roi_html_render_with_pos_xpath` returns the raw HTML for a specific ROI, while `get_roi_text_render_with_pos_xpath` generates a concise text snippet for debugging or LLM consumption.

## Classifying Regions of Interest via LLM

Once the DOM structure is established, Spider Creator evaluates each ROI against the user's planning intent using the `classify_roi_html_create_cand_spider` function in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py). This component iterates over `dom_repr.tree_regions_system.sorted_roi_by_pos_xpath` and prepares classification prompts.

For each ROI, the system retrieves both HTML and text renders. The text render is printed for debugging purposes, while the HTML render is sent to the LLM using a structured output schema defined by the `HTMLClassificationResult` Pydantic model. The prompt (`ROI_CLASSIFICATION_PROMPT`) embeds both the ROI HTML and the planning JSON, requesting a boolean `result` field and an explanatory string.

The LLM invocation occurs through the shared instance `shared.o3_llm` with the `with_structured_output` method to enforce response conformity. Classification outcomes and optional generated spider code are stored in a results dictionary keyed by ROI index.

```python
from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider
import json

planning_json = json.dumps({
    "target": "product_price",
    "attributes": ["currency", "amount"]
})

results = classify_roi_html_create_cand_spider(
    dom_repr=dom_repr,
    extracted_content_on_rec="price data",
    planning=planning_json,
    max_exec_amt=50  # Limits LLM calls per execution

)

# Process classification results

for idx, outcome in results.items():
    if outcome and outcome.result:
        print(f"ROI {idx} matches planning intent")
        if outcome.spider_code:
            print("Generated candidate spider:")
            print(outcome.spider_code)

```

## Complete Workflow: From Raw HTML to Classified ROIs

The following example demonstrates the integrated pipeline, combining DOM construction and ROI classification as implemented in the main [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) workflow.

```python

# Stage 1: Build DOM representation

from pipeline.make_dom_repr import make_dom_representation

website_html = recordings_data[0]["website_html"]
dom_repr = make_dom_representation(
    website_html=website_html,
    MAX_NODE_REPR_LENGTH=32768,
)

# Stage 2: Classify against planning intent

from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider

planning = {
    "target": "article_content",
    "fields": ["headline", "author", "publish_date"]
}

CAND_SPIDER_CREATION_RESULTS = classify_roi_html_create_cand_spider(
    dom_repr=dom_repr,
    extracted_content_on_rec="article",
    planning=json.dumps(planning),
)

# Results contain classification booleans and optional spider code

print(f"Processed {len(CAND_SPIDER_CREATION_RESULTS)} ROIs")

```

After classification, valid candidates proceed to [`pipeline/make_candsp_runnable.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_candsp_runnable.py), which transforms the classified results into executable Scrapy spider code.

## Summary

- **DOM Representation**: The `make_dom_representation` function in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) utilizes the **betterhtmlchunking** library to convert raw HTML into a navigable `DomRepresentation` object with configurable size limits.
- **ROI Discovery**: The `tree_regions_system` identifies regions of interest and sorts them by XPath position, accessible via `sorted_roi_by_pos_xpath`.
- **Dual Rendering**: The `render_system` provides both HTML and text representations of ROIs through `get_roi_html_render_with_pos_xpath` and `get_roi_text_render_with_pos_xpath`.
- **LLM Classification**: The `classify_roi_html_create_cand_spider` function in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) evaluates each ROI against planning JSON using the `shared.o3_llm` instance with structured output schemas.
- **Structured Output**: The `HTMLClassificationResult` Pydantic model ensures consistent boolean results and explanatory text from the LLM classification process.

## Frequently Asked Questions

### What is a Region of Interest (ROI) in Spider Creator?

In Spider Creator, a **Region of Interest (ROI)** represents a discrete fragment of the DOM tree identified by the `tree_regions_system` as a potential candidate for data extraction. These regions are sorted by their XPath position and evaluated individually against the user's planning intent to determine relevance for spider generation.

### How does the DomRepresentation object manage large HTML documents?

The `DomRepresentation` object handles large documents through the `MAX_NODE_REPR_LENGTH` parameter passed to `make_dom_representation` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py). This configurable limit (defaulting to 32768 characters) controls the maximum size of individual node representations during the chunking process, preventing memory issues while preserving structural integrity.

### What LLM model does Spider Creator use for ROI classification?

According to the source code in [`shared.py`](https://github.com/carlosplanchon/spidercreator/blob/main/shared.py), Spider Creator uses the `o3_llm` shared instance for ROI classification. The system invokes this model through the `with_structured_output` method in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py), enforcing compliance with the `HTMLClassificationResult` Pydantic schema for reliable boolean classification results.

### How does the structured output schema ensure reliable classification results?

The `HTMLClassificationResult` Pydantic model defines a strict schema requiring the LLM to return a boolean `result` field and a string `explanation` field. By using `with_structured_output` on the `shared.o3_llm` instance, the system in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) constrains the model to produce parseable, type-safe responses rather than free-form text, enabling deterministic downstream processing of classification outcomes.