How to Customize the create_spider() Function for Advanced Use Cases in SpiderCreator

You can customize the create_spider() function by extending the RecordingAPIContext class, wrapping the recording loop with custom hooks, or modifying the pipeline stages in the pipeline/ directory to inject custom logic, alternative LLMs, or post-processing steps.

The create_spider() function in SpiderCreator serves as the high-level entry point that orchestrates browser recording and spider generation. While the default implementation in main.py handles standard workflows, you can customize the create_spider() function to support advanced scenarios like custom authentication, timeout enforcement, or alternative language models. This guide demonstrates how to extend each subsystem without breaking the core pipeline.

Understanding the create_spider() Architecture

Before customizing, you must understand the three subsystems that create_spider() ties together. Each subsystem is isolated in its own module, allowing you to override specific components.

Task ID and Recording Folder

The function first generates a UUID and creates a per-run folder to store artifacts. This logic lives in main.py lines 13-24 (main.py#L13-L24).

Browser-Use Recording API

The Recording API runs receive_bu_data.py inside a PTY, exposing a local HTTP endpoint that receives the browser-use prompt and stores recorded activity. The context manager RecordingAPIContext and the runner run_recorder_with_pty() are defined in exec_funcs.py lines 84-108 and 24-33 respectively (exec_funcs.py#L84-L108, exec_funcs.py#L24-L33).

Spider-Creation Pipeline

Once recording finishes, the pipeline builds a Scrapy draft, refines it, validates execution, and merges candidates into a runnable Playwright spider. Key functions include:

• make_scrapy_spider_draft() in pipeline/spider_draft.py ([spider_draft.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py))
• classify_roi_html_create_cand_spider() in pipeline/roiclf_spcandmkr.py ([roiclf_spcandmkr.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py))
• get_spider_combination() in pipeline/sp_combination.py ([sp_combination.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py))

Customization Strategies for create_spider()

Because each step is isolated in its own function or class, you can plug in custom logic without touching the core pipeline.

Pass Additional Parameters to the Recording API

When you need to change the port, add authentication tokens, or inject extra query-string arguments to receive_bu_data.py, extend RecordingAPIContext and override __enter__.

# custom_api.py

from exec_funcs import RecordingAPIContext, start_recording_api_thread

class AuthenticatedAPIContext(RecordingAPIContext):
    def __init__(self, token: str, **kwargs):
        super().__init__(**kwargs)
        self.token = token

    def __enter__(self):
        def on_spawn(proc):
            self._process = proc
            # Send authentication token right after the process starts

            proc.write(f"Authorization: Bearer {self.token}\n".encode())

        self._thread = start_recording_api_thread(
            port=self.port,
            folder_name=self.folder_name,
            on_spawn=on_spawn,
        )
        return self

Then swap the import in main.py:

from custom_api import AuthenticatedAPIContext as RecordingAPIContext

Source: exec_funcs.py defines the original context – see lines 84-108 (exec_funcs.py#L84-L108).

Hook Into the Recording Loop

To perform side-effects while browser-use data is being recorded—such as logging progress to a database, enforcing a timeout, or streaming PTY output elsewhere—wrap run_recorder_with_pty().

def run_recorder_with_pty_custom(api_port: int, task: str, on_line=None) -> str:
    from exec_funcs import run_recorder_with_pty
    output = run_recorder_with_pty(api_port, task)   # existing implementation

    for line in output.splitlines():
        if on_line:
            on_line(line)               # user-provided callback

    return output

Pass a lambda when calling create_spider():

run_recorder_with_pty_custom(
    api_port=api_port,
    task=browser_use_task,
    on_line=lambda l: my_logger.info(f"REC: {l}")
)

Source: Core recorder lives in exec_funcs.py – lines 24-33 (exec_funcs.py#L24-L33).

Insert Pre-Processing or Post-Processing Steps

When you need to transform the browser_use_task before it reaches the API—such as injecting placeholders—or post-process the generated spider code to add custom middlewares, wrap the call inside create_spider().

def preprocess_task(task: str) -> str:
    # Example: replace {{DATE}} with today's ISO date

    from datetime import date
    return task.replace("{{DATE}}", date.today().isoformat())

def postprocess_spider(code: str) -> str:
    # Append a custom middleware import

    middleware = "\nfrom myproject.middlewares import MyMiddleware\n"
    return code + middleware

# In main.py (replace the original body)

task_prepared = preprocess_task(browser_use_task)
run_recorder_with_pty(api_port=api_port, task=task_prepared)

spider_output = run_spider_creator_with_pty(task_id=task_id)
final_spider = postprocess_spider(spider_output)

# Save or return `final_spider` as needed

Source: create_spider() body is in main.py lines 13-47 (main.py#L13-L47).

Change the Underlying Pipeline (LLM or Prompt)

To replace the default GPT-4o model, inject extra system messages, or skip certain pipeline stages, modify the pipeline orchestration in spidercreator.py. The pipeline calls these key functions:

Function	Role
make_scrapy_spider_draft (pipeline/spider_draft.py)	Generates a first-draft Scrapy spider from the recorded plan.
classify_roi_html_create_cand_spider (pipeline/roiclf_spcandmkr.py)	Splits the plan into candidate spiders.
execute_cand_spiders (ctxexec/exec_sp.py + related)	Runs each candidate spider in a sandbox.
run_verification_on_cand_spider_exec_results (pipeline/verification_pipeline.py)	Verifies execution using XPath checks.
get_spider_combination (pipeline/sp_combination.py)	Merges candidates into a final Playwright spider.

To swap the LLM, edit the LLM initialization in the respective pipeline file. For example, pipeline/spider_draft.py contains a gpt4o_llm variable. Replace it with your own model wrapper or adjust the system prompt.

Source examples:

• Draft creation – pipeline/spider_draft.py line 31 (def make_scrapy_spider_draft).
• Combination – pipeline/sp_combination.py line 51 (def get_spider_combination_prompt).

Direct links: [spider_draft.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py), [sp_combination.py](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py).

Persist or Export Intermediate Artifacts

When you need the raw candidate spider code, execution logs, or verification scores for auditing, read the results/{task_id} folder after run_spider_creator_with_pty finishes. The folder already contains:

• recordings/ – raw browser-use recordings.
• spider_code.py – final combined spider.
• pipeline_logs/ (if you enable logging in the pipeline) – add a logging configuration to each pipeline module to write JSON logs to results/{task_id}/pipeline_logs/.

You can expose a helper in exec_funcs.py:

def load_intermediate_results(task_id: str) -> dict:
    import json, pathlib
    base = pathlib.Path(f"recordings/{task_id}")
    return {
        "draft": (base / "draft.txt").read_text(),
        "candidates": json.loads((base / "candidates.json").read_text()),
        "verification": json.loads((base / "verification.json").read_text()),
    }

Practical Code Examples

Adding a Timeout to the Recording Phase

To enforce a maximum duration for the browser-use recording, wrap the recorder in a threaded timeout.

import threading, time, pathlib
from exec_funcs import RecordingAPIContext, run_recorder_with_pty, run_spider_creator_with_pty, generate_task_id

def create_spider_with_timeout(browser_use_task: str,
                               api_port: int = 9000,
                               timeout: int = 120) -> str:
    task_id = generate_task_id()
    folder = f"recordings/{task_id}"
    pathlib.Path(folder).mkdir(parents=True, exist_ok=True)

    with RecordingAPIContext(port=api_port, folder_name=folder):
        # Run the recorder in a separate thread so we can enforce a timeout

        recorder_thread = threading.Thread(
            target=run_recorder_with_pty,
            args=(api_port, browser_use_task),
            daemon=True,
        )
        recorder_thread.start()
        recorder_thread.join(timeout)          # <-- timeout enforcement

        if recorder_thread.is_alive():
            print("[WARN] Recording timed out – proceeding anyway")
        else:
            print("[INFO] Recording finished within limit")

    # Continue with the usual pipeline

    run_spider_creator_with_pty(task_id=task_id)
    return task_id

References: RecordingAPIContext – exec_funcs.py‑108; run_recorder_with_pty – exec_funcs.py‑33.

Injecting Custom Middleware into the Final Spider

To append custom middleware or imports to the generated spider, post-process the output file.

import pathlib
from main import create_spider

def create_spider_with_middleware(browser_use_task: str,
                                  api_port: int = 9000,
                                  middleware_code: str = "") -> str:
    task_id = create_spider(browser_use_task, api_port)   # uses default flow

    final_path = pathlib.Path(f"recordings/{task_id}/spider_code.py")
    spider_code = final_path.read_text()

    # Append user-provided middleware import and activation

    if middleware_code:
        spider_code += "\n" + middleware_code + "\n"
        final_path.write_text(spider_code)

    print(f"Custom spider saved to {final_path}")
    return task_id

References: Final spider is written in spidercreator.py (invoked by run_spider_creator_with_pty). The file path logic is in exec_funcs.py‑77.

Key Files for create_spider() Customization

File	Role	Direct link
main.py	Public wrapper create_spider() – orchestrates recording + pipeline.	https://github.com/carlosplanchon/spidercreator/blob/main/main.py
exec_funcs.py	Low-level PTY helpers, task-id generation, RecordingAPIContext, and the two run_*_with_pty utilities.	https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py
pipeline/spider_draft.py	Generates the initial Scrapy draft from the recorded plan.	https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py
pipeline/roiclf_spcandmkr.py	Splits the plan into region-of-interest candidate spiders.	https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py
pipeline/verification_pipeline.py	Runs XPath-based verification on each candidate execution.	https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py
pipeline/sp_combination.py	Merges verified candidates into a single Playwright spider.	https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py
ctxexec/exec_sp.py & related ctxexec/*	Executes candidate spiders in isolated environments.	https://github.com/carlosplanchon/spidercreator/tree/main/ctxexec
examples/*.py	Ready-to-run demonstrations that call create_spider().	https://github.com/carlosplanchon/spidercreator/tree/main/examples

Summary

• create_spider() in main.py orchestrates three isolated subsystems: task ID generation, the browser-use recording API, and the spider-creation pipeline.
• You can pass additional parameters to the recording API by subclassing RecordingAPIContext from exec_funcs.py and overriding __enter__.
• Hook into the recording loop by wrapping run_recorder_with_pty() with custom callbacks or threading logic to enforce timeouts or stream logs.
• Insert pre-processing or post-processing steps by wrapping the task string before it reaches the API or by modifying the final spider code after run_spider_creator_with_pty() completes.
• Change the underlying pipeline by editing LLM initializations in pipeline/spider_draft.py or pipeline/sp_combination.py to use different models or prompts.
• Persist intermediate artifacts by reading from the results/{task_id} folder or adding logging configurations to pipeline modules.

Frequently Asked Questions

How do I change the port used by the Recording API when calling create_spider()?

The default port is hardcoded in RecordingAPIContext. To customize it, subclass RecordingAPIContext from exec_funcs.py and pass your desired port to the superclass constructor. Then use your subclass in place of the original context manager inside main.py.

Can I use a different LLM model instead of GPT-4o in the spider generation pipeline?

Yes. The pipeline modules initialize LLM variables locally. For example, pipeline/spider_draft.py contains a gpt4o_llm variable used by make_scrapy_spider_draft(). Replace this initialization with your own model wrapper or API client, ensuring it conforms to the expected interface for generating spider drafts.

Where are the intermediate candidate spiders stored during execution?

Intermediate artifacts are written to the results/{task_id} directory. This includes the recordings/ subfolder for raw browser-use data and spider_code.py for the final output. You can enable additional JSON logging in pipeline modules to write candidate spiders and verification scores to results/{task_id}/pipeline_logs/ for auditing purposes.

Is it possible to add a timeout to the browser recording phase?

Yes. Because run_recorder_with_pty() blocks until the recording finishes, you can wrap it in a Python threading.Thread and use thread.join(timeout=seconds) to enforce a limit. If the thread is still alive after the timeout, you can proceed with partial data or raise an exception, depending on your use case.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}