# How to Customize the create_spider() Function for Advanced Use Cases in SpiderCreator

> Learn to customize the create_spider() function in SpiderCreator for advanced use cases. Extend RecordingAPIContext, use custom hooks, or modify pipeline stages for custom logic and LLMs.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: how-to-guide
- Published: 2026-02-26

---

**You can customize the `create_spider()` function by extending the `RecordingAPIContext` class, wrapping the recording loop with custom hooks, or modifying the pipeline stages in the `pipeline/` directory to inject custom logic, alternative LLMs, or post-processing steps.**

The `create_spider()` function in **SpiderCreator** serves as the high-level entry point that orchestrates browser recording and spider generation. While the default implementation in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) handles standard workflows, you can customize the `create_spider()` function to support advanced scenarios like custom authentication, timeout enforcement, or alternative language models. This guide demonstrates how to extend each subsystem without breaking the core pipeline.

## Understanding the create_spider() Architecture

Before customizing, you must understand the three subsystems that `create_spider()` ties together. Each subsystem is isolated in its own module, allowing you to override specific components.

### Task ID and Recording Folder

The function first generates a **UUID** and creates a per-run folder to store artifacts. This logic lives in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) lines 13-24 ([`main.py#L13-L24`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py#L13-L24)).

### Browser-Use Recording API

The **Recording API** runs [`receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/receive_bu_data.py) inside a PTY, exposing a local HTTP endpoint that receives the browser-use prompt and stores recorded activity. The context manager `RecordingAPIContext` and the runner `run_recorder_with_pty()` are defined in [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) lines 84-108 and 24-33 respectively ([`exec_funcs.py#L84-L108`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L84-L108), [`exec_funcs.py#L24-L33`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L24-L33)).

### Spider-Creation Pipeline

Once recording finishes, the pipeline builds a Scrapy draft, refines it, validates execution, and merges candidates into a runnable Playwright spider. Key functions include:

- `make_scrapy_spider_draft()` in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) ([[`spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_draft.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py))
- `classify_roi_html_create_cand_spider()` in [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) ([[`roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/roiclf_spcandmkr.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py))
- `get_spider_combination()` in [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) ([[`sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/sp_combination.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py))

## Customization Strategies for create_spider()

Because each step is isolated in its own function or class, you can plug in custom logic without touching the core pipeline.

### Pass Additional Parameters to the Recording API

When you need to change the port, add authentication tokens, or inject extra query-string arguments to [`receive_bu_data.py`](https://github.com/carlosplanchon/spidercreator/blob/main/receive_bu_data.py), extend `RecordingAPIContext` and override `__enter__`.

```python

# custom_api.py

from exec_funcs import RecordingAPIContext, start_recording_api_thread

class AuthenticatedAPIContext(RecordingAPIContext):
    def __init__(self, token: str, **kwargs):
        super().__init__(**kwargs)
        self.token = token

    def __enter__(self):
        def on_spawn(proc):
            self._process = proc
            # Send authentication token right after the process starts

            proc.write(f"Authorization: Bearer {self.token}\n".encode())

        self._thread = start_recording_api_thread(
            port=self.port,
            folder_name=self.folder_name,
            on_spawn=on_spawn,
        )
        return self

```

Then swap the import in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py):

```python
from custom_api import AuthenticatedAPIContext as RecordingAPIContext

```

*Source:* [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) defines the original context – see lines 84-108 ([`exec_funcs.py#L84-L108`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L84-L108)).

### Hook Into the Recording Loop

To perform side-effects while browser-use data is being recorded—such as logging progress to a database, enforcing a timeout, or streaming PTY output elsewhere—wrap `run_recorder_with_pty()`.

```python
def run_recorder_with_pty_custom(api_port: int, task: str, on_line=None) -> str:
    from exec_funcs import run_recorder_with_pty
    output = run_recorder_with_pty(api_port, task)   # existing implementation

    for line in output.splitlines():
        if on_line:
            on_line(line)               # user-provided callback

    return output

```

Pass a lambda when calling `create_spider()`:

```python
run_recorder_with_pty_custom(
    api_port=api_port,
    task=browser_use_task,
    on_line=lambda l: my_logger.info(f"REC: {l}")
)

```

*Source:* Core recorder lives in [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) – lines 24-33 ([`exec_funcs.py#L24-L33`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L24-L33)).

### Insert Pre-Processing or Post-Processing Steps

When you need to transform the `browser_use_task` before it reaches the API—such as injecting placeholders—or post-process the generated spider code to add custom middlewares, wrap the call inside `create_spider()`.

```python
def preprocess_task(task: str) -> str:
    # Example: replace {{DATE}} with today's ISO date

    from datetime import date
    return task.replace("{{DATE}}", date.today().isoformat())

def postprocess_spider(code: str) -> str:
    # Append a custom middleware import

    middleware = "\nfrom myproject.middlewares import MyMiddleware\n"
    return code + middleware

# In main.py (replace the original body)

task_prepared = preprocess_task(browser_use_task)
run_recorder_with_pty(api_port=api_port, task=task_prepared)

spider_output = run_spider_creator_with_pty(task_id=task_id)
final_spider = postprocess_spider(spider_output)

# Save or return `final_spider` as needed

```

*Source:* `create_spider()` body is in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) lines 13-47 ([`main.py#L13-L47`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py#L13-L47)).

### Change the Underlying Pipeline (LLM or Prompt)

To replace the default GPT-4o model, inject extra system messages, or skip certain pipeline stages, modify the pipeline orchestration in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py). The pipeline calls these key functions:

| Function | Role |
|----------|------|
| `make_scrapy_spider_draft` ([`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py)) | Generates a first-draft Scrapy spider from the recorded plan. |
| `classify_roi_html_create_cand_spider` ([`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py)) | Splits the plan into candidate spiders. |
| `execute_cand_spiders` ([`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) + related) | Runs each candidate spider in a sandbox. |
| `run_verification_on_cand_spider_exec_results` ([`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py)) | Verifies execution using XPath checks. |
| `get_spider_combination` ([`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py)) | Merges candidates into a final Playwright spider. |

To swap the LLM, edit the LLM initialization in the respective pipeline file. For example, [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) contains a `gpt4o_llm` variable. Replace it with your own model wrapper or adjust the system prompt.

*Source examples:*  
- Draft creation – [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) line 31 (`def make_scrapy_spider_draft`).  
- Combination – [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) line 51 (`def get_spider_combination_prompt`).  

Direct links: [[`spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_draft.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py), [[`sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/sp_combination.py)](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py).

### Persist or Export Intermediate Artifacts

When you need the raw candidate spider code, execution logs, or verification scores for auditing, read the `results/{task_id}` folder after `run_spider_creator_with_pty` finishes. The folder already contains:

- `recordings/` – raw browser-use recordings.  
- [`spider_code.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_code.py) – final combined spider.  
- `pipeline_logs/` (if you enable logging in the pipeline) – add a `logging` configuration to each pipeline module to write JSON logs to `results/{task_id}/pipeline_logs/`.

You can expose a helper in [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py):

```python
def load_intermediate_results(task_id: str) -> dict:
    import json, pathlib
    base = pathlib.Path(f"recordings/{task_id}")
    return {
        "draft": (base / "draft.txt").read_text(),
        "candidates": json.loads((base / "candidates.json").read_text()),
        "verification": json.loads((base / "verification.json").read_text()),
    }

```

## Practical Code Examples

### Adding a Timeout to the Recording Phase

To enforce a maximum duration for the browser-use recording, wrap the recorder in a threaded timeout.

```python
import threading, time, pathlib
from exec_funcs import RecordingAPIContext, run_recorder_with_pty, run_spider_creator_with_pty, generate_task_id

def create_spider_with_timeout(browser_use_task: str,
                               api_port: int = 9000,
                               timeout: int = 120) -> str:
    task_id = generate_task_id()
    folder = f"recordings/{task_id}"
    pathlib.Path(folder).mkdir(parents=True, exist_ok=True)

    with RecordingAPIContext(port=api_port, folder_name=folder):
        # Run the recorder in a separate thread so we can enforce a timeout

        recorder_thread = threading.Thread(
            target=run_recorder_with_pty,
            args=(api_port, browser_use_task),
            daemon=True,
        )
        recorder_thread.start()
        recorder_thread.join(timeout)          # <-- timeout enforcement

        if recorder_thread.is_alive():
            print("[WARN] Recording timed out – proceeding anyway")
        else:
            print("[INFO] Recording finished within limit")

    # Continue with the usual pipeline

    run_spider_creator_with_pty(task_id=task_id)
    return task_id

```

*References:* `RecordingAPIContext` – [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L84)‑108; `run_recorder_with_pty` – [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L24)‑33.

### Injecting Custom Middleware into the Final Spider

To append custom middleware or imports to the generated spider, post-process the output file.

```python
import pathlib
from main import create_spider

def create_spider_with_middleware(browser_use_task: str,
                                  api_port: int = 9000,
                                  middleware_code: str = "") -> str:
    task_id = create_spider(browser_use_task, api_port)   # uses default flow

    final_path = pathlib.Path(f"recordings/{task_id}/spider_code.py")
    spider_code = final_path.read_text()

    # Append user-provided middleware import and activation

    if middleware_code:
        spider_code += "\n" + middleware_code + "\n"
        final_path.write_text(spider_code)

    print(f"Custom spider saved to {final_path}")
    return task_id

```

*References:* Final spider is written in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) (invoked by `run_spider_creator_with_pty`). The file path logic is in [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py#L73)‑77.

## Key Files for create_spider() Customization

| File | Role | Direct link |
|------|------|-------------|
| [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) | Public wrapper `create_spider()` – orchestrates recording + pipeline. | <https://github.com/carlosplanchon/spidercreator/blob/main/main.py> |
| [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) | Low-level PTY helpers, task-id generation, `RecordingAPIContext`, and the two `run_*_with_pty` utilities. | <https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py> |
| [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) | Generates the initial Scrapy draft from the recorded plan. | <https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py> |
| [`pipeline/roiclf_spcandmkr.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py) | Splits the plan into region-of-interest candidate spiders. | <https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/roiclf_spcandmkr.py> |
| [`pipeline/verification_pipeline.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py) | Runs XPath-based verification on each candidate execution. | <https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/verification_pipeline.py> |
| [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) | Merges verified candidates into a single Playwright spider. | <https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py> |
| [`ctxexec/exec_sp.py`](https://github.com/carlosplanchon/spidercreator/blob/main/ctxexec/exec_sp.py) & related `ctxexec/*` | Executes candidate spiders in isolated environments. | <https://github.com/carlosplanchon/spidercreator/tree/main/ctxexec> |
| `examples/*.py` | Ready-to-run demonstrations that call `create_spider()`. | <https://github.com/carlosplanchon/spidercreator/tree/main/examples> |

## Summary

- **`create_spider()`** in [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py) orchestrates three isolated subsystems: task ID generation, the browser-use recording API, and the spider-creation pipeline.
- You can **pass additional parameters** to the recording API by subclassing `RecordingAPIContext` from [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) and overriding `__enter__`.
- **Hook into the recording loop** by wrapping `run_recorder_with_pty()` with custom callbacks or threading logic to enforce timeouts or stream logs.
- **Insert pre-processing or post-processing** steps by wrapping the task string before it reaches the API or by modifying the final spider code after `run_spider_creator_with_pty()` completes.
- **Change the underlying pipeline** by editing LLM initializations in [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) or [`pipeline/sp_combination.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/sp_combination.py) to use different models or prompts.
- **Persist intermediate artifacts** by reading from the `results/{task_id}` folder or adding logging configurations to pipeline modules.

## Frequently Asked Questions

### How do I change the port used by the Recording API when calling create_spider()?

The default port is hardcoded in `RecordingAPIContext`. To customize it, subclass `RecordingAPIContext` from [`exec_funcs.py`](https://github.com/carlosplanchon/spidercreator/blob/main/exec_funcs.py) and pass your desired port to the superclass constructor. Then use your subclass in place of the original context manager inside [`main.py`](https://github.com/carlosplanchon/spidercreator/blob/main/main.py).

### Can I use a different LLM model instead of GPT-4o in the spider generation pipeline?

Yes. The pipeline modules initialize LLM variables locally. For example, [`pipeline/spider_draft.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/spider_draft.py) contains a `gpt4o_llm` variable used by `make_scrapy_spider_draft()`. Replace this initialization with your own model wrapper or API client, ensuring it conforms to the expected interface for generating spider drafts.

### Where are the intermediate candidate spiders stored during execution?

Intermediate artifacts are written to the `results/{task_id}` directory. This includes the `recordings/` subfolder for raw browser-use data and [`spider_code.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spider_code.py) for the final output. You can enable additional JSON logging in pipeline modules to write candidate spiders and verification scores to `results/{task_id}/pipeline_logs/` for auditing purposes.

### Is it possible to add a timeout to the browser recording phase?

Yes. Because `run_recorder_with_pty()` blocks until the recording finishes, you can wrap it in a Python `threading.Thread` and use `thread.join(timeout=seconds)` to enforce a limit. If the thread is still alive after the timeout, you can proceed with partial data or raise an exception, depending on your use case.