Recommended DOM Chunk Size and ROI Thresholds in Spider Creator

The recommended DOM chunk size is 32,768 characters (MAX_NODE_REPR_LENGTH) and the ROI threshold is 75 regions (max_exec_amt), balancing memory efficiency with granular region detection while keeping LLM classification costs manageable.

The Spider Creator pipeline compresses raw HTML into chunked DOM representations before scanning for actionable Regions of Interest (ROIs). Properly configuring the DOM chunk size and ROI thresholds ensures the system processes large commercial websites without triggering memory pressure or excessive LLM API calls.

Understanding the Core Parameters

The Spider Creator repository exposes two critical tuning knobs that control how HTML is processed and how many regions receive LLM classification.

MAX_NODE_REPR_LENGTH (DOM Chunk Size)

The MAX_NODE_REPR_LENGTH parameter defines the maximum character length of each node’s string representation before the DOM splits into chunks. This value is passed to make_dom_representation in pipeline/make_dom_representation.py【/cache/repos/github.com/carlosplanchon/spidercreator/main/pipeline/make_dom_representation.py#L9-L15】 and defaults to 32768 characters.

Larger values reduce chunk count and API overhead but increase memory consumption. Smaller values create finer-grained chunks that improve ROI precision on massive pages at the cost of higher processing overhead.

max_exec_amt (ROI Threshold)

The max_exec_amt parameter sets an upper bound on how many ROIs the system classifies and converts into candidate spiders. This prevents runaway LLM calls on pages containing hundreds of potential regions. The default value is 75, defined in the function signature of classify_roi_html_create_cand_spider within pipeline/roiclf_spcandmkr.py【/cache/repos/github.com/carlosplanchon/spidercreator/main/pipeline/roiclf_spcandmkr.py#L86-L88】.

Why These Defaults Work

The default DOM chunk size of 32768 represents twice the "safe" 16,384 character limit used by the upstream betterhtmlchunking library. This doubling accommodates typical commercial pages (hundreds of kilobytes of HTML) without fragmenting semantic structures unnaturally【/cache/repos/github.com/carlosplanchon/spidercreator/main/spidercreator.py#L155-L160】.

The default ROI threshold of 75 caps LLM classification requests to keep latency and token costs predictable. Most realistic websites contain significantly fewer than 75 actionable sections, making this limit sufficient for broad coverage while preventing cost explosions on unusually dense pages【/cache/repos/github.com/carlosplanchon/spidercreator/main/spidercreator.py#L176-L182】.

Configuring the DOM Chunk Size

Adjust MAX_NODE_REPR_LENGTH when calling make_dom_representation to tune the granularity versus memory tradeoff:

from pipeline.make_dom_repr import make_dom_representation

# Increase to 64KB for very large pages to reduce chunk count

dom_repr = make_dom_representation(
    website_html=html_source,
    MAX_NODE_REPR_LENGTH=65536   # ← increased from default 32768

)

Decrease this value to 16384 if you observe memory pressure during processing, or increase it to 65536 or higher to minimize API calls on massive single-page applications.

Adjusting the ROI Threshold

Modify the max_exec_amt argument when invoking classify_roi_html_create_cand_spider to control how many regions receive LLM classification:

from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider

# Raise limit for dense documentation sites with many extractable sections

candidate_spiders = classify_roi_html_create_cand_spider(
    dom_repr=dom_repr,
    extracted_content_on_rec=extracted_text,
    planning=plan_json,
    max_exec_amt=120   # ← increased from default 75

)

Lowering this threshold to 50 or below reduces costs for exploratory runs, while raising it risks higher token consumption without necessarily improving extraction quality.

Complete Pipeline Configuration

Combine both parameters in the main orchestration flow to fine-tune the Spider Creator pipeline for your specific target:

from pipeline.make_dom_repr import make_dom_representation
from pipeline.roiclf_spcandmkr import classify_roi_html_create_cand_spider

# Step 1: Chunk DOM with custom size

dom = make_dom_representation(
    website_html=recording["website_html"],
    MAX_NODE_REPR_LENGTH=32768   # adjust for memory/granularity needs

)

# Step 2: Classify ROIs with custom threshold

candidates = classify_roi_html_create_cand_spider(
    dom_repr=dom,
    extracted_content_on_rec=recording["extracted_content"],
    planning=plan_json,
    max_exec_amt=75   # adjust based on expected page density and budget

)

Summary

• Default DOM chunk size: 32,768 characters (MAX_NODE_REPR_LENGTH) provides optimal balance for typical commercial HTML.
• Default ROI threshold: 75 regions (max_exec_amt) prevents runaway LLM costs while covering most actionable page sections.
• Configuration location: Set chunk size in pipeline/make_dom_representation.py and ROI limits in pipeline/roiclf_spcandmkr.py.
• Tuning guidance: Increase chunk sizes to reduce API calls on massive pages; decrease ROI thresholds to minimize classification costs on budget-constrained runs.

Frequently Asked Questions

What happens if I increase the DOM chunk size beyond 32,768?

Increasing MAX_NODE_REPR_LENGTH reduces the number of chunks generated, which lowers API call overhead but increases memory consumption per chunk. For pages exceeding several megabytes of HTML, values above 65536 may cause memory pressure depending on your runtime environment.

Why is the ROI threshold limited to 75 by default?

The default limit of 75 ROIs prevents excessive LLM classification calls on pages containing hundreds of potential regions. According to the Spider Creator source code, most realistic websites contain far fewer actionable sections, making 75 sufficient for broad coverage while maintaining predictable token costs【/cache/repos/github.com/carlosplanchon/spidercreator/main/pipeline/roiclf_spcandmkr.py#L86-L88】.

Can I set these parameters via environment variables?

The current implementation requires explicit parameter passing in Python. You must adjust MAX_NODE_REPR_LENGTH in the make_dom_representation call and max_exec_amt when invoking classify_roi_html_create_cand_spider rather than using environment-based configuration.

How do these settings affect processing speed?

Larger DOM chunks reduce the number of LLM calls required for the initial compression phase but may slow down ROI detection if chunks become too granular. Conversely, lower ROI thresholds speed up classification by capping the number of regions processed, though you may miss viable extraction targets on dense pages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}