How to Configure the API Port and Set Custom Task IDs in Spider Creator

You configure the API port using the --port argument in receive_bu_data.py (default 8000) or record_activity.py (default 9000), and set custom task IDs using the --task_id argument in spidercreator.py to isolate recordings and results.

Spider Creator is an open-source automation framework that records browser interactions via FastAPI endpoints and generates Scrapy spiders from those recordings. To deploy it in production or multi-tenant environments, you must configure the API port for the recording server and use unique task identifiers to organize outputs. This guide explains exactly how to configure the API port and set custom task IDs in Spider Creator based on the source code in the carlosplanchon/spidercreator repository.

Configuring the API Port in Spider Creator

Spider Creator uses two separate CLI entry points that expose --port arguments. The recorder client and the FastAPI server must agree on the same port to successfully transmit agent history.

Setting the Port in the FastAPI Server (receive_bu_data.py)

The recording server is defined in receive_bu_data.py and defaults to port 8000 on lines 60-63.

# receive_bu_data.py#L60-L63

parser.add_argument(
    "--port",
    type=int,
    default=8000,
    help="Port to run the server on"
)

To run the server on a custom port, pass the desired integer:

python receive_bu_data.py --port 9100 --folder_name recordings

Configuring the Recorder Client Port (record_activity.py)

The agent recorder in record_activity.py defaults to port 9000 on lines 18-24. This is the destination port for HTTP POST requests to /post_agent_history_step.

# record_activity.py#L18-L24

parser.add_argument(
    "--port",
    type=int,
    default=9000,
    help="Port of the API server"
)

Launch the recorder against your custom server port:

python record_activity.py --port 9100 --task "Extract product listings from example.com"

Using the Helper Functions in exec_funcs.py

For programmatic control, exec_funcs.py provides PTY-based helpers that forward port arguments. Lines 32-44 wrap the recording API:

# exec_funcs.py#L32-L44

def run_recording_api_with_pty(
    port: int = 8000,
    folder_name: str = "recordings"
) -> threading.Thread:
    """Run the recording API in a PTY thread."""
    return run_script_with_pty(
        "receive_bu_data.py",
        f"--port {port} --folder_name {folder_name}"
    )

And lines 77-93 handle the recorder client:

# exec_funcs.py#L77-L93

def run_recorder_with_pty(
    api_port: int = 8000,
    task: str = ""
) -> str:
    """Run the recorder in a PTY and return the output."""
    return run_script_with_pty(
        "record_activity.py",
        f"--port {api_port} --task '{task}'"
    )

Setting Custom Task IDs in Spider Creator

Task IDs isolate recordings and generated spiders into separate directories, preventing data collisions between concurrent runs.

The --task_id Argument in spidercreator.py

The main pipeline in spidercreator.py requires a --task_id argument defined on lines 47-52:

# spidercreator.py#L47-L52

parser.add_argument(
    "--task_id",
    type=str,
    required=True,
    help="Task ID for the spider creation process"
)

Spider Creator uses this value to read from recordings/<task_id>/ and write results to results/<task_id>/.

Run the pipeline with a custom identifier:

python spidercreator.py --task_id my-custom-task

Programmatic Task ID Assignment

The helper in exec_funcs.py (lines 77-93) also supports task IDs for the main pipeline:

# exec_funcs.py#L77-L93 (spidercreator variant)

def run_spider_creator_with_pty(task_id: str) -> str:
    """Run spidercreator.py in a PTY and return the output."""
    return run_script_with_pty(
        "spidercreator.py",
        f"--task_id {task_id}"
    )

Invoke it programmatically:

from exec_funcs import run_spider_creator_with_pty

spider_output = run_spider_creator_with_pty(task_id="my-run-001")
print(spider_output)

Complete Workflow Example

This end-to-end example demonstrates configuring a non-default API port and a custom task ID:

from exec_funcs import (
    run_recording_api_with_pty,
    run_recorder_with_pty,
    run_spider_creator_with_pty,
)

# 1️⃣ Start the recording API on port 9100

api_thread = run_recording_api_with_pty(
    port=9100,
    folder_name="recordings"
)

# 2️⃣ Record browser activity against port 9100

run_recorder_with_pty(
    api_port=9100,
    task="Scrape real-estate listings"
)

# 3️⃣ Define a unique task ID

task_id = "realestate-2024-08"

# 4️⃣ Generate the spider using the custom task ID

spider_output = run_spider_creator_with_pty(task_id=task_id)
print(spider_output)

Summary

• API Port Configuration: The FastAPI server (receive_bu_data.py) defaults to port 8000, while the recorder client (record_activity.py) defaults to port 9000. Pass --port <integer> to either script to override these values.
• Custom Task IDs: The spidercreator.py pipeline requires --task_id <string> to isolate recordings in recordings/<task_id>/ and output results to results/<task_id>/.
• Programmatic Control: Use the PTY helpers in exec_funcs.py (run_recording_api_with_pty, run_recorder_with_pty, run_spider_creator_with_pty) to manage ports and task IDs without shell commands.

Frequently Asked Questions

What is the default API port for Spider Creator's recording server?

The FastAPI recording server in receive_bu_data.py defaults to port 8000, while the agent recorder in record_activity.py defaults to port 9000. These defaults are defined in the argument parsers on lines 60-63 and 18-24 respectively.

Can I run multiple Spider Creator instances with different task IDs simultaneously?

Yes. Because Spider Creator uses the --task_id argument to isolate data into separate directories (recordings/<task_id>/ and results/<task_id>/), you can run multiple pipelines concurrently without file collisions, provided each instance uses a unique task identifier.

How does the task ID affect file storage in Spider Creator?

The task ID determines the subdirectory paths for both input and output. The pipeline reads recorded browser steps from recordings/<task_id>/ and writes generated spider code and results to results/<task_id>/, as enforced by the required --task_id argument in spidercreator.py on lines 47-52.

What happens if I don't specify a custom port for the recording API?

If you omit the --port argument, the system falls back to the hardcoded defaults: 8000 for the server (receive_bu_data.py) and 9000 for the recorder (record_activity.py). Mismatched ports between these components will cause the recorder's POST requests to /post_agent_history_step to fail with connection errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}