How to Tune Performance for Large-Scale Web Scraping Projects with SpiderCreator

Tune performance for large-scale web scraping projects by optimizing DOM chunk sizes, limiting ROI classification scope, reducing verification delays, and parallelizing candidate execution across the SpiderCreator pipeline.

SpiderCreator is an open-source framework that transforms recorded browser sessions into production-ready Playwright spiders through a multi-stage pipeline. When you need to tune performance for large-scale web scraping projects, understanding the seven core pipeline stages—from recording ingestion to final spider assembly—allows you to target specific bottlenecks and reduce CPU, memory, and API overhead.

Understanding the SpiderCreator Pipeline Architecture

The pipeline orchestrated in spidercreator.py processes data through distinct stages, each offering specific tuning opportunities:

Stage	Component	Performance Lever
1. Load recordings	load_recordings in utils/recordings.py	Strip unused fields like model_outputs before ingestion
2. Filter recordings	RecordingInterpreter in planning/rec_filtering.py	Prune unnecessary fields such as model_thoughts
3. Build DOM representation	make_dom_representation in pipeline/make_dom_repr.py	Adjust MAX_NODE_REPR_LENGTH (default 32,768)
4. ROI classification	classify_roi_html_create_cand_spider in pipeline/roiclf_spcandmkr.py	Limit max_exec_amt (default 75)
5. Execute candidates	execute_cand_spiders in ctxexec/pipeline.py	Control max_exec_instances (default 20) and enable multiprocessing
6. Verification	run_verification_on_cand_spider_exec_results in pipeline/verification_pipeline.py	Reduce or remove the 5-second sleep delay
7. Final assembly	get_spider_combination in pipeline/sp_combination.py	Minimal overhead; keep lightweight

Stage-by-Stage Performance Optimization Strategies

Stage 1: Optimize Recording Loading

The load_recordings function in utils/recordings.py reads JSON-encoded browser logs. Pre-process your recordings to remove fields you never reference, such as model_outputs, to reduce memory footprint and parsing time.

Stage 2: Filter Recordings Efficiently

RecordingInterpreter in planning/rec_filtering.py automatically discards about:blank pages and noise. You can further optimize by explicitly removing fields like model_thoughts that are not required for your specific spider generation plan.

Stage 3: Tune DOM Chunking

In pipeline/make_dom_repr.py, the make_dom_representation function uses betterhtmlchunking to segment HTML. The MAX_NODE_REPR_LENGTH parameter defaults to 32,768 characters.

• Increase this value only if you require richer context for complex pages, at the cost of higher memory usage.
• Decrease this value to reduce processing time and memory pressure for each chunk.

Stage 4: Limit ROI Classification Scope

The classify_roi_html_create_cand_spider function in pipeline/roiclf_spcandmkr.py uses an LLM to classify regions of interest (ROIs). The max_exec_amt parameter (default 75) controls how many ROIs are examined.

Lowering this value prevents the LLM from processing thousands of tiny snippets, reducing API costs and latency.

Stage 5: Optimize Candidate Execution

execute_cand_spiders in ctxexec/pipeline.py spins up temporary servers for each candidate. Three key optimizations apply here:

1. Reduce max_exec_instances from the default 20 to avoid exhausting CPU and port ranges.
2. Implement multiprocessing to run independent recordings in parallel.
3. Reuse browser contexts instead of launching a new Playwright instance per candidate.

Stage 6: Reduce Verification Delays

In pipeline/verification_pipeline.py, the run_verification_on_cand_spider_exec_results function includes a 5-second sleep (time.sleep(5)) per candidate to avoid rate limits.

Reducing this to 0.5 seconds or removing it entirely dramatically speeds up the verification loop, though you should implement alternative rate limiting for LLM or external APIs.

Stage 7: Final Assembly Considerations

The get_spider_combination function in pipeline/sp_combination.py merges selected candidates into the final spider. This stage involves minimal computation; ensure it remains lightweight by avoiding unnecessary data transformations.

Advanced Performance Techniques

Beyond pipeline-specific tuning, implement these cross-cutting strategies to tune performance for large-scale web scraping projects:

• Batch LLM calls – Both ROI classification and verification stages invoke LLMs. If your endpoint supports batching (OpenAI-compatible APIs), wrap multiple ROI prompts into single requests to reduce network overhead.
• Cache HTML and LLM responses – Persist HTML chunks and LLM completions to disk using pickle or JSON. When re-running the pipeline, load from cache instead of recomputing.
• Run headless, low-resource browsers – Configure Playwright with headless=True and block unnecessary resources (images, CSS, fonts) using page.route to minimize network I/O and memory usage.
• Parallelize recording processing – Since each task_id operates on independent directories (recordings/<task_id> and results/<task_id>), launch separate processes for each top-level spidercreator.main() call.
• Profile the pipeline – Use Python’s cProfile or pyinstrument around the main loop in spidercreator.py to identify which stage dominates runtime for your specific target sites.

Implementation Examples

Lower the ROI Classification Limit

Reduce the number of regions the LLM examines by adjusting max_exec_amt in pipeline/roiclf_spcandmkr.py:

def classify_roi_html_create_cand_spider(
    dom_repr,
    extracted_content_on_rec: str,
    planning: str,
    max_exec_amt: int = 30,  # Reduced from default 75

):
    ...

Reduce Verification Sleep Time

Speed up the verification loop by lowering the delay in pipeline/verification_pipeline.py:

def run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec,
    verification_criteria,
):
    for key, cand_spider_executor in CAND_SPIDER_EXEC_RESULTS.items():
        time.sleep(0.5)  # Reduced from 5 seconds

        ...

Increase DOM Chunk Size

For pages requiring richer context, adjust MAX_NODE_REPR_LENGTH in pipeline/make_dom_repr.py:

def make_dom_representation(
    website_html: str,
    MAX_NODE_REPR_LENGTH: int = 65536,  # Increased to 64KB

) -> DomRepresentation:
    ...

Parallelize Candidate Execution

Implement multiprocessing in ctxexec/pipeline.py to run candidates concurrently:

from multiprocessing import Pool

def _run_one_candidate(args):
    key, chunk, recordings_data, port_offset = args
    executor = CandSpiderExecutor(
        spider_code=chunk.spider_code,
        recordings_data=recordings_data,
        port_offset=port_offset,
    )
    executor.start()
    return key, executor, executor.port_offset

def execute_cand_spiders(
    CAND_SPIDER_CREATION_RESULTS,
    recordings_data,
    max_exec_instances: int = 20,
):
    pool = Pool(processes=4)  # Parallel workers

    tasks = []
    port_offset = 0
    
    for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
        if chunk and chunk.spider_code:
            tasks.append((key, chunk, recordings_data, port_offset))
            port_offset += 1

    results = pool.map(_run_one_candidate, tasks)
    pool.close()
    pool.join()

    return {k: exec_obj for k, exec_obj, _ in results}

Disable Image Loading in Playwright

Reduce resource consumption by blocking unnecessary assets in your generated spiders:

from playwright.sync_api import sync_playwright

def launch_browser():
    p = sync_playwright().start()
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()
    
    # Block images, fonts, and stylesheets

    context.route(
        "**/*.{png,jpg,jpeg,svg,css,woff,woff2}", 
        lambda route: route.abort()
    )
    return browser, context

Summary

To tune performance for large-scale web scraping projects using SpiderCreator, focus on these high-impact optimizations:

• Reduce LLM overhead by lowering max_exec_amt in pipeline/roiclf_spcandmkr.py and batching API calls where possible.
• Speed up verification by reducing the 5-second sleep in pipeline/verification_pipeline.py to 0.5 seconds or less.
• Control memory usage by adjusting MAX_NODE_REPR_LENGTH in pipeline/make_dom_repr.py based on your context requirements.
• Parallelize execution using multiprocessing in ctxexec/pipeline.py and process independent task_id recordings in separate processes.
• Optimize browser resources by running Playwright in headless mode with images, CSS, and fonts blocked.

Frequently Asked Questions

How do I reduce API costs when running SpiderCreator on hundreds of sites?

Limit the number of regions the LLM must classify by setting max_exec_amt to 30 or lower in pipeline/roiclf_spcandmkr.py. Additionally, implement a caching layer using pickle or JSON to store LLM responses and HTML chunks, preventing redundant API calls when reprocessing similar pages.

What is the fastest way to speed up the verification stage?

The verification loop in pipeline/verification_pipeline.py contains a hardcoded 5-second sleep between candidates. Reduce this value to 0.5 seconds or remove it entirely to dramatically accelerate processing. If you remove the delay, implement alternative rate limiting for your LLM provider to avoid hitting API quotas.

How can I prevent memory exhaustion when processing large HTML documents?

Adjust the MAX_NODE_REPR_LENGTH parameter in pipeline/make_dom_repr.py. The default value of 32,768 characters provides rich context but consumes significant memory. Lower this value to reduce per-chunk memory footprint, or increase it only when processing complex pages that require extended context for accurate spider generation.

Is it safe to run multiple SpiderCreator tasks in parallel?

Yes. Each task_id operates within isolated directories (recordings/<task_id> and results/<task_id>) with no shared state. You can safely launch separate processes for each top-level spidercreator.main() call, or implement multiprocessing within ctxexec/pipeline.py to execute candidate spiders concurrently across multiple CPU cores.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}