How Spider Creator Handles DOM Representation and ROI Classification: A Technical Deep Dive

Spider Creator converts raw HTML into a navigable DOM tree using the betterhtmlchunking library, then employs an LLM with structured output schemas to classify each Region of Interest (ROI) against user planning intent.

The carlosplanchon/spidercreator repository implements a two-stage pipeline for intelligent web scraping. This article examines how the system builds a structured DOM representation from raw HTML and subsequently performs ROI classification to identify relevant page fragments for automatic spider generation.

Building the DOM Representation with betterhtmlchunking

Spider Creator constructs a DomRepresentation object by processing raw HTML through the betterhtmlchunking library. The entry point is the make_dom_representation function in pipeline/make_dom_repr.py, which accepts a configurable MAX_NODE_REPR_LENGTH parameter to handle large documents.

The function wraps the HTML and initializes internal chunking via dom_repr.start(). The resulting object exposes two critical subsystems: the tree_regions_system, which discovers and sorts ROIs by their XPath position, and the render_system, which converts ROIs into consumable formats.

from pipeline.make_dom_repr import make_dom_representation

dom_repr = make_dom_representation(
    website_html=website_html,
    MAX_NODE_REPR_LENGTH=32768,  # Configurable limit for node representation size

)

# Access sorted ROIs by XPath position

roi_count = len(dom_repr.tree_regions_system.sorted_roi_by_pos_xpath)
print(f"Discovered {roi_count} potential regions of interest")

The render_system provides two key methods: get_roi_html_render_with_pos_xpath returns the raw HTML for a specific ROI, while get_roi_text_render_with_pos_xpath generates a concise text snippet for debugging or LLM consumption.

Classifying Regions of Interest via LLM

Once the DOM structure is established, Spider Creator evaluates each ROI against the user's planning intent using the classify_roi_html_create_cand_spider function in pipeline/roiclf_spcandmkr.py. This component iterates over dom_repr.tree_regions_system.sorted_roi_by_pos_xpath and prepares classification prompts.

For each ROI, the system retrieves both HTML and text renders. The text render is printed for debugging purposes, while the HTML render is sent to the LLM using a structured output schema defined by the HTMLClassificationResult Pydantic model. The prompt (ROI_CLASSIFICATION_PROMPT) embeds both the ROI HTML and the planning JSON, requesting a boolean result field and an explanatory string.

The LLM invocation occurs through the shared instance shared.o3_llm with the with_structured_output method to enforce response conformity. Classification outcomes and optional generated spider code are stored in a results dictionary keyed by ROI index.

from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider
import json

planning_json = json.dumps({
    "target": "product_price",
    "attributes": ["currency", "amount"]
})

results = classify_roi_html_create_cand_spider(
    dom_repr=dom_repr,
    extracted_content_on_rec="price data",
    planning=planning_json,
    max_exec_amt=50  # Limits LLM calls per execution

)

# Process classification results

for idx, outcome in results.items():
    if outcome and outcome.result:
        print(f"ROI {idx} matches planning intent")
        if outcome.spider_code:
            print("Generated candidate spider:")
            print(outcome.spider_code)

Complete Workflow: From Raw HTML to Classified ROIs

The following example demonstrates the integrated pipeline, combining DOM construction and ROI classification as implemented in the main spidercreator.py workflow.

# Stage 1: Build DOM representation

from pipeline.make_dom_repr import make_dom_representation

website_html = recordings_data[0]["website_html"]
dom_repr = make_dom_representation(
    website_html=website_html,
    MAX_NODE_REPR_LENGTH=32768,
)

# Stage 2: Classify against planning intent

from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider

planning = {
    "target": "article_content",
    "fields": ["headline", "author", "publish_date"]
}

CAND_SPIDER_CREATION_RESULTS = classify_roi_html_create_cand_spider(
    dom_repr=dom_repr,
    extracted_content_on_rec="article",
    planning=json.dumps(planning),
)

# Results contain classification booleans and optional spider code

print(f"Processed {len(CAND_SPIDER_CREATION_RESULTS)} ROIs")

After classification, valid candidates proceed to pipeline/make_candsp_runnable.py, which transforms the classified results into executable Scrapy spider code.

Summary

• DOM Representation: The make_dom_representation function in pipeline/make_dom_repr.py utilizes the betterhtmlchunking library to convert raw HTML into a navigable DomRepresentation object with configurable size limits.
• ROI Discovery: The tree_regions_system identifies regions of interest and sorts them by XPath position, accessible via sorted_roi_by_pos_xpath.
• Dual Rendering: The render_system provides both HTML and text representations of ROIs through get_roi_html_render_with_pos_xpath and get_roi_text_render_with_pos_xpath.
• LLM Classification: The classify_roi_html_create_cand_spider function in pipeline/roiclf_spcandmkr.py evaluates each ROI against planning JSON using the shared.o3_llm instance with structured output schemas.
• Structured Output: The HTMLClassificationResult Pydantic model ensures consistent boolean results and explanatory text from the LLM classification process.

Frequently Asked Questions

What is a Region of Interest (ROI) in Spider Creator?

In Spider Creator, a Region of Interest (ROI) represents a discrete fragment of the DOM tree identified by the tree_regions_system as a potential candidate for data extraction. These regions are sorted by their XPath position and evaluated individually against the user's planning intent to determine relevance for spider generation.

How does the DomRepresentation object manage large HTML documents?

The DomRepresentation object handles large documents through the MAX_NODE_REPR_LENGTH parameter passed to make_dom_representation in pipeline/make_dom_repr.py. This configurable limit (defaulting to 32768 characters) controls the maximum size of individual node representations during the chunking process, preventing memory issues while preserving structural integrity.

What LLM model does Spider Creator use for ROI classification?

According to the source code in shared.py, Spider Creator uses the o3_llm shared instance for ROI classification. The system invokes this model through the with_structured_output method in pipeline/roiclf_spcandmkr.py, enforcing compliance with the HTMLClassificationResult Pydantic schema for reliable boolean classification results.

How does the structured output schema ensure reliable classification results?

The HTMLClassificationResult Pydantic model defines a strict schema requiring the LLM to return a boolean result field and a string explanation field. By using with_structured_output on the shared.o3_llm instance, the system in pipeline/roiclf_spcandmkr.py constrains the model to produce parseable, type-safe responses rather than free-form text, enabling deterministic downstream processing of classification outcomes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}