# How to Tune Performance for Large-Scale Web Scraping Projects with SpiderCreator

> Optimize large-scale web scraping performance with SpiderCreator. Learn strategies for DOM chunking ROI scope verification delays and parallel execution.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: performance
- Published: 2026-02-26

---

**Tune performance for large-scale web scraping projects by optimizing DOM chunk sizes, limiting ROI classification scope, reducing verification delays, and parallelizing candidate execution across the SpiderCreator pipeline.**

SpiderCreator is an open-source framework that transforms recorded browser sessions into production-ready Playwright spiders through a multi-stage pipeline. When you need to tune performance for large-scale web scraping projects, understanding the seven core pipeline stages—from recording ingestion to final spider assembly—allows you to target specific bottlenecks and reduce CPU, memory, and API overhead.

## Understanding the SpiderCreator Pipeline Architecture

The pipeline orchestrated in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) processes data through distinct stages, each offering specific tuning opportunities:

| Stage | Component | Performance Lever |
|-------|-----------|-----------------|
| 1. Load recordings | `load_recordings` in [`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py) | Strip unused fields like `model_outputs` before ingestion |
| 2. Filter recordings | `RecordingInterpreter` in [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py) | Prune unnecessary fields such as `model_thoughts` |
| 3. Build DOM representation | `make_dom_representation` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) | Adjust `MAX_NODE_REPR_LENGTH` (default 32,768) |
| 4. ROI classification | `classify_roi_html_create_cand_spider` in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) | Limit `max_exec_amt` (default 75) |
| 5. Execute candidates | `execute_cand_spiders` in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) | Control `max_exec_instances` (default 20) and enable multiprocessing |
| 6. Verification | `run_verification_on_cand_spider_exec_results` in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) | Reduce or remove the 5-second sleep delay |
| 7. Final assembly | `get_spider_combination` in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) | Minimal overhead; keep lightweight |

## Stage-by-Stage Performance Optimization Strategies

### Stage 1: Optimize Recording Loading

The `load_recordings` function in [`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py) reads JSON-encoded browser logs. Pre-process your recordings to remove fields you never reference, such as `model_outputs`, to reduce memory footprint and parsing time.

### Stage 2: Filter Recordings Efficiently

`RecordingInterpreter` in [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py) automatically discards `about:blank` pages and noise. You can further optimize by explicitly removing fields like `model_thoughts` that are not required for your specific spider generation plan.

### Stage 3: Tune DOM Chunking

In [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py), the `make_dom_representation` function uses **betterhtmlchunking** to segment HTML. The `MAX_NODE_REPR_LENGTH` parameter defaults to 32,768 characters.

- **Increase** this value only if you require richer context for complex pages, at the cost of higher memory usage.
- **Decrease** this value to reduce processing time and memory pressure for each chunk.

### Stage 4: Limit ROI Classification Scope

The `classify_roi_html_create_cand_spider` function in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) uses an LLM to classify regions of interest (ROIs). The `max_exec_amt` parameter (default 75) controls how many ROIs are examined.

Lowering this value prevents the LLM from processing thousands of tiny snippets, reducing API costs and latency.

### Stage 5: Optimize Candidate Execution

`execute_cand_spiders` in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) spins up temporary servers for each candidate. Three key optimizations apply here:

1. **Reduce `max_exec_instances`** from the default 20 to avoid exhausting CPU and port ranges.
2. **Implement multiprocessing** to run independent recordings in parallel.
3. **Reuse browser contexts** instead of launching a new Playwright instance per candidate.

### Stage 6: Reduce Verification Delays

In [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py), the `run_verification_on_cand_spider_exec_results` function includes a **5-second sleep** (`time.sleep(5)`) per candidate to avoid rate limits.

Reducing this to 0.5 seconds or removing it entirely dramatically speeds up the verification loop, though you should implement alternative rate limiting for LLM or external APIs.

### Stage 7: Final Assembly Considerations

The `get_spider_combination` function in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) merges selected candidates into the final spider. This stage involves minimal computation; ensure it remains lightweight by avoiding unnecessary data transformations.

## Advanced Performance Techniques

Beyond pipeline-specific tuning, implement these cross-cutting strategies to tune performance for large-scale web scraping projects:

- **Batch LLM calls** – Both ROI classification and verification stages invoke LLMs. If your endpoint supports batching (OpenAI-compatible APIs), wrap multiple ROI prompts into single requests to reduce network overhead.
- **Cache HTML and LLM responses** – Persist HTML chunks and LLM completions to disk using `pickle` or JSON. When re-running the pipeline, load from cache instead of recomputing.
- **Run headless, low-resource browsers** – Configure Playwright with `headless=True` and block unnecessary resources (images, CSS, fonts) using `page.route` to minimize network I/O and memory usage.
- **Parallelize recording processing** – Since each `task_id` operates on independent directories (`recordings/<task_id>` and `results/<task_id>`), launch separate processes for each top-level `spidercreator.main()` call.
- **Profile the pipeline** – Use Python’s `cProfile` or `pyinstrument` around the main loop in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) to identify which stage dominates runtime for your specific target sites.

## Implementation Examples

### Lower the ROI Classification Limit

Reduce the number of regions the LLM examines by adjusting `max_exec_amt` in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py):

```python
def classify_roi_html_create_cand_spider(
    dom_repr,
    extracted_content_on_rec: str,
    planning: str,
    max_exec_amt: int = 30,  # Reduced from default 75

):
    ...

```

### Reduce Verification Sleep Time

Speed up the verification loop by lowering the delay in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py):

```python
def run_verification_on_cand_spider_exec_results(
    CAND_SPIDER_EXEC_RESULTS,
    extracted_content_on_rec,
    verification_criteria,
):
    for key, cand_spider_executor in CAND_SPIDER_EXEC_RESULTS.items():
        time.sleep(0.5)  # Reduced from 5 seconds

        ...

```

### Increase DOM Chunk Size

For pages requiring richer context, adjust `MAX_NODE_REPR_LENGTH` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py):

```python
def make_dom_representation(
    website_html: str,
    MAX_NODE_REPR_LENGTH: int = 65536,  # Increased to 64KB

) -> DomRepresentation:
    ...

```

### Parallelize Candidate Execution

Implement multiprocessing in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) to run candidates concurrently:

```python
from multiprocessing import Pool

def _run_one_candidate(args):
    key, chunk, recordings_data, port_offset = args
    executor = CandSpiderExecutor(
        spider_code=chunk.spider_code,
        recordings_data=recordings_data,
        port_offset=port_offset,
    )
    executor.start()
    return key, executor, executor.port_offset

def execute_cand_spiders(
    CAND_SPIDER_CREATION_RESULTS,
    recordings_data,
    max_exec_instances: int = 20,
):
    pool = Pool(processes=4)  # Parallel workers

    tasks = []
    port_offset = 0
    
    for key, chunk in CAND_SPIDER_CREATION_RESULTS.items():
        if chunk and chunk.spider_code:
            tasks.append((key, chunk, recordings_data, port_offset))
            port_offset += 1

    results = pool.map(_run_one_candidate, tasks)
    pool.close()
    pool.join()

    return {k: exec_obj for k, exec_obj, _ in results}

```

### Disable Image Loading in Playwright

Reduce resource consumption by blocking unnecessary assets in your generated spiders:

```python
from playwright.sync_api import sync_playwright

def launch_browser():
    p = sync_playwright().start()
    browser = p.chromium.launch(headless=True)
    context = browser.new_context()
    
    # Block images, fonts, and stylesheets

    context.route(
        "**/*.{png,jpg,jpeg,svg,css,woff,woff2}", 
        lambda route: route.abort()
    )
    return browser, context

```

## Summary

To tune performance for large-scale web scraping projects using SpiderCreator, focus on these high-impact optimizations:

- **Reduce LLM overhead** by lowering `max_exec_amt` in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) and batching API calls where possible.
- **Speed up verification** by reducing the 5-second sleep in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) to 0.5 seconds or less.
- **Control memory usage** by adjusting `MAX_NODE_REPR_LENGTH` in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py) based on your context requirements.
- **Parallelize execution** using multiprocessing in [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) and process independent `task_id` recordings in separate processes.
- **Optimize browser resources** by running Playwright in headless mode with images, CSS, and fonts blocked.

## Frequently Asked Questions

### How do I reduce API costs when running SpiderCreator on hundreds of sites?

Limit the number of regions the LLM must classify by setting `max_exec_amt` to 30 or lower in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py). Additionally, implement a caching layer using `pickle` or JSON to store LLM responses and HTML chunks, preventing redundant API calls when reprocessing similar pages.

### What is the fastest way to speed up the verification stage?

The verification loop in [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) contains a hardcoded 5-second sleep between candidates. Reduce this value to 0.5 seconds or remove it entirely to dramatically accelerate processing. If you remove the delay, implement alternative rate limiting for your LLM provider to avoid hitting API quotas.

### How can I prevent memory exhaustion when processing large HTML documents?

Adjust the `MAX_NODE_REPR_LENGTH` parameter in [`pipeline/make_dom_repr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/make_dom_repr.py). The default value of 32,768 characters provides rich context but consumes significant memory. Lower this value to reduce per-chunk memory footprint, or increase it only when processing complex pages that require extended context for accurate spider generation.

### Is it safe to run multiple SpiderCreator tasks in parallel?

Yes. Each `task_id` operates within isolated directories (`recordings/<task_id>` and `results/<task_id>`) with no shared state. You can safely launch separate processes for each top-level `spidercreator.main()` call, or implement multiprocessing within [`ctxexec/pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/pipeline.py) to execute candidate spiders concurrently across multiple CPU cores.