How Mindmap Generation Works in Spider Creator: A Technical Deep Dive

Spider Creator generates visual Mermaid mindmaps by feeding filtered browser recordings through a LangChain LLM pipeline that enforces structured output via Pydantic validation.

Mindmap generation serves as the visual foundation of the Spider Creator architecture, transforming raw browsing recordings into hierarchical Mermaid diagrams that map automated navigation workflows. This process bridges the gap between recorded browser actions and executable web scraping logic, providing both human-readable visualization and structured data for downstream pipeline stages.

The Mindmap Generation Pipeline

The architecture implements a six-stage pipeline that converts raw recordings into valid Mermaid syntax:

1. Load and filter recordings – Raw browser actions are ingested from recordings/<task_id> and processed by RecordingInterpreter to remove noise and standardize the action sequence.

2. Pass to generator – The filtered list is handed to make_mermaid_mindmap in pipeline/mindmap.py.

3. Structured LLM binding – A LangChain chat model (o3_llm) is wrapped with a Pydantic schema (Mermaid) to enforce strict output validation.

4. Prompt composition – The system prompt MERMAID_WORK_MINDMAP_PROMPT provides few-shot examples and syntax rules.

5. LLM invocation – The function constructs HumanMessage and SystemMessage objects, then calls mermaid_structured_llm.invoke().

6. Extract Mermaid code – The validated Mermaid object returns the final mermaid_code string for rendering.

Core Components and File Structure

pipeline/mindmap.py

This module contains the central orchestration logic for mindmap generation. It defines the Mermaid Pydantic model (lines 12-16) that enforces output structure, the detailed system prompt spanning lines 21-66, and the make_mermaid_mindmap function that coordinates LLM invocation (lines 70-77).

spidercreator.py

The main entry point orchestrates the end-to-end workflow. Lines 69-77 handle recording loading and filtering via RecordingInterpreter, while lines 82-84 invoke make_mermaid_mindmap and output the resulting diagram.

Supporting Modules

• shared.py – Instantiates the shared o3_llm LangChain model used across all pipeline stages.
• utils/recordings.py – Provides load_recordings helper for ingesting raw browser action JSON from disk.
• planning/rec_filtering.py – Implements RecordingInterpreter for cleaning and structuring raw recordings before mindmap generation.

Step-by-Step Implementation Details

Recording Ingestion and Filtering

Before mindmap generation begins, raw browser recordings undergo strict preprocessing. The RecordingInterpreter class (implemented in planning/rec_filtering.py) loads actions from the recordings/<task_id> directory and filters out redundant events, standardizes URL formats, and sequences navigation steps. This ensures the LLM receives only relevant, structured data rather than raw browser noise.

Structured LLM Output with Pydantic

Spider Creator enforces output validity through structured generation. In pipeline/mindmap.py, the code binds the o3_llm LangChain model to a Pydantic schema:

class Mermaid(BaseModel):
    mermaid_code: str

This binding (lines 12-16) guarantees that the LLM returns valid JSON containing a mermaid_code field, eliminating syntax errors and malformed diagrams before they reach downstream components.

Prompt Engineering for Mermaid Syntax

The system prompt MERMAID_WORK_MINDMAP_PROMPT (lines 21-66) provides comprehensive instructions for generating valid Mermaid mindmap syntax. It specifies:

• Hierarchical node structure using root and child identifiers
• Step numbering conventions for action sequences
• URL preservation requirements
• Action type categorization (click, input, navigation)
• Few-shot examples demonstrating correct indentation and syntax

This prompt engineering ensures the LLM outputs diagrams that are both syntactically valid and semantically meaningful for web automation workflows.

Practical Usage Examples

Minimal Standalone Implementation

You can generate mindmaps directly using the core pipeline function:

from pipeline.mindmap import make_mermaid_mindmap
from utils.recordings import load_recordings

# Load filtered recordings from disk

recordings = load_recordings("recordings/example_task")

# Generate Mermaid mindmap via LLM

mermaid_code = make_mermaid_mindmap(recordings)

print("=== Generated Mermaid Mindmap ===")
print(mermaid_code)

This example demonstrates the direct interface between raw browser recordings and the LLM-powered visualization generator.

Integration in the Main Spider Creator Workflow

The primary orchestration in spidercreator.py shows how mindmap generation fits into the broader automation pipeline:

# Load and filter recordings via interpreter

recordings_itpr = RecordingInterpreter(recordings_path)
filtered_recordings = recordings_itpr.get_filtered_recordings_list()

# Generate mindmap through LLM pipeline

mermaid_code: str = make_mermaid_mindmap(
    recordings=filtered_recordings
)

print("\n--- MERMAID WORK MINDMAP ---")
print(mermaid_code)

This integration ensures that only cleaned, relevant browser actions are visualized, providing a clear hierarchical map for subsequent spider code generation stages.

Summary

• Spider Creator transforms raw browser recordings into visual Mermaid mindmaps through a structured LLM pipeline.
• The process centers on make_mermaid_mindmap in pipeline/mindmap.py, which enforces output validation via Pydantic schemas.
• LangChain and the o3_llm model handle generation, guided by the detailed MERMAID_WORK_MINDMAP_PROMPT system prompt.
• Input data flows through RecordingInterpreter in planning/rec_filtering.py to ensure clean, structured action sequences.
• The resulting Mermaid code provides hierarchical visualization of automated browsing workflows for downstream spider generation.

Frequently Asked Questions

What is the purpose of mindmap generation in Spider Creator?

Mindmap generation creates a hierarchical visual representation of recorded browser actions, transforming raw navigation data into structured Mermaid diagrams. This visualization serves as an intermediate artifact that helps both human operators understand the browsing flow and downstream automation components generate precise web scraping logic.

Which LLM does Spider Creator use for generating Mermaid diagrams?

Spider Creator utilizes the o3_llm LangChain chat model, which is instantiated in shared.py and shared across pipeline components. This model is specifically wrapped with a Pydantic Mermaid schema in pipeline/mindmap.py to enforce structured output and ensure valid Mermaid syntax generation.

How does Spider Creator ensure valid Mermaid syntax output?

The architecture enforces validity through structured generation using Pydantic models. The make_mermaid_mindmap function binds the LLM to a Mermaid schema that requires a mermaid_code string field. Additionally, the MERMAID_WORK_MINDMAP_PROMPT provides detailed syntax rules, step numbering conventions, and few-shot examples to guide the LLM toward syntactically correct output.

Can I customize the mindmap prompt for specific browsing workflows?

Yes, you can modify the MERMAID_WORK_MINDMAP_PROMPT constant defined in pipeline/mindmap.py (lines 21-66) to customize the mindmap structure, node hierarchy, or action categorization. When adjusting the prompt, ensure you maintain the Pydantic schema constraints defined in the Mermaid class to prevent output validation errors during LLM invocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}