# How Mindmap Generation Works in Spider Creator: A Technical Deep Dive

> Explore Spider Creator's technical deep dive on generative mindmaps. Understand how browser recordings, LangChain LLM, and Pydantic create structured visual Mermaid mindmaps.

- Repository: [Carlos A. Planchón/spidercreator](https://github.com/carlosplanchon/spidercreator)
- Tags: deep-dive
- Published: 2026-02-26

---

**Spider Creator generates visual Mermaid mindmaps by feeding filtered browser recordings through a LangChain LLM pipeline that enforces structured output via Pydantic validation.**

Mindmap generation serves as the visual foundation of the Spider Creator architecture, transforming raw browsing recordings into hierarchical Mermaid diagrams that map automated navigation workflows. This process bridges the gap between recorded browser actions and executable web scraping logic, providing both human-readable visualization and structured data for downstream pipeline stages.

## The Mindmap Generation Pipeline

The architecture implements a six-stage pipeline that converts raw recordings into valid Mermaid syntax:

1. **Load and filter recordings** – Raw browser actions are ingested from `recordings/<task_id>` and processed by `RecordingInterpreter` to remove noise and standardize the action sequence.

2. **Pass to generator** – The filtered list is handed to `make_mermaid_mindmap` in [`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py).

3. **Structured LLM binding** – A LangChain chat model (`o3_llm`) is wrapped with a Pydantic schema (`Mermaid`) to enforce strict output validation.

4. **Prompt composition** – The system prompt `MERMAID_WORK_MINDMAP_PROMPT` provides few-shot examples and syntax rules.

5. **LLM invocation** – The function constructs `HumanMessage` and `SystemMessage` objects, then calls `mermaid_structured_llm.invoke()`.

6. **Extract Mermaid code** – The validated `Mermaid` object returns the final `mermaid_code` string for rendering.

## Core Components and File Structure

### pipeline/mindmap.py

This module contains the central orchestration logic for mindmap generation. It defines the `Mermaid` Pydantic model (lines 12-16) that enforces output structure, the detailed system prompt spanning lines 21-66, and the `make_mermaid_mindmap` function that coordinates LLM invocation (lines 70-77).

### spidercreator.py

The main entry point orchestrates the end-to-end workflow. Lines 69-77 handle recording loading and filtering via `RecordingInterpreter`, while lines 82-84 invoke `make_mermaid_mindmap` and output the resulting diagram.

### Supporting Modules

- **[`shared.py`](https://github.com/carlosplanchon/spidercreator/blob/main/shared.py)** – Instantiates the shared `o3_llm` LangChain model used across all pipeline stages.
- **[`utils/recordings.py`](https://github.com/carlosplanchon/spidercreator/blob/main/utils/recordings.py)** – Provides `load_recordings` helper for ingesting raw browser action JSON from disk.
- **[`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py)** – Implements `RecordingInterpreter` for cleaning and structuring raw recordings before mindmap generation.

## Step-by-Step Implementation Details

### Recording Ingestion and Filtering

Before mindmap generation begins, raw browser recordings undergo strict preprocessing. The `RecordingInterpreter` class (implemented in [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py)) loads actions from the `recordings/<task_id>` directory and filters out redundant events, standardizes URL formats, and sequences navigation steps. This ensures the LLM receives only relevant, structured data rather than raw browser noise.

### Structured LLM Output with Pydantic

Spider Creator enforces output validity through **structured generation**. In [`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py), the code binds the `o3_llm` LangChain model to a Pydantic schema:

```python
class Mermaid(BaseModel):
    mermaid_code: str

```

This binding (lines 12-16) guarantees that the LLM returns valid JSON containing a `mermaid_code` field, eliminating syntax errors and malformed diagrams before they reach downstream components.

### Prompt Engineering for Mermaid Syntax

The system prompt `MERMAID_WORK_MINDMAP_PROMPT` (lines 21-66) provides comprehensive instructions for generating valid Mermaid mindmap syntax. It specifies:

- Hierarchical node structure using `root` and child identifiers
- Step numbering conventions for action sequences
- URL preservation requirements
- Action type categorization (click, input, navigation)
- Few-shot examples demonstrating correct indentation and syntax

This prompt engineering ensures the LLM outputs diagrams that are both syntactically valid and semantically meaningful for web automation workflows.

## Practical Usage Examples

### Minimal Standalone Implementation

You can generate mindmaps directly using the core pipeline function:

```python
from pipeline.mindmap import make_mermaid_mindmap
from utils.recordings import load_recordings

# Load filtered recordings from disk

recordings = load_recordings("recordings/example_task")

# Generate Mermaid mindmap via LLM

mermaid_code = make_mermaid_mindmap(recordings)

print("=== Generated Mermaid Mindmap ===")
print(mermaid_code)

```

This example demonstrates the direct interface between raw browser recordings and the LLM-powered visualization generator.

### Integration in the Main Spider Creator Workflow

The primary orchestration in [`spidercreator.py`](https://github.com/carlosplanchon/spidercreator/blob/main/spidercreator.py) shows how mindmap generation fits into the broader automation pipeline:

```python

# Load and filter recordings via interpreter

recordings_itpr = RecordingInterpreter(recordings_path)
filtered_recordings = recordings_itpr.get_filtered_recordings_list()

# Generate mindmap through LLM pipeline

mermaid_code: str = make_mermaid_mindmap(
    recordings=filtered_recordings
)

print("\n--- MERMAID WORK MINDMAP ---")
print(mermaid_code)

```

This integration ensures that only cleaned, relevant browser actions are visualized, providing a clear hierarchical map for subsequent spider code generation stages.

## Summary

- **Spider Creator** transforms raw browser recordings into visual Mermaid mindmaps through a structured LLM pipeline.
- The process centers on `make_mermaid_mindmap` in [`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py), which enforces output validation via Pydantic schemas.
- **LangChain** and the `o3_llm` model handle generation, guided by the detailed `MERMAID_WORK_MINDMAP_PROMPT` system prompt.
- Input data flows through `RecordingInterpreter` in [`planning/rec_filtering.py`](https://github.com/carlosplanchon/spidercreator/blob/main/planning/rec_filtering.py) to ensure clean, structured action sequences.
- The resulting Mermaid code provides hierarchical visualization of automated browsing workflows for downstream spider generation.

## Frequently Asked Questions

### What is the purpose of mindmap generation in Spider Creator?

Mindmap generation creates a hierarchical visual representation of recorded browser actions, transforming raw navigation data into structured Mermaid diagrams. This visualization serves as an intermediate artifact that helps both human operators understand the browsing flow and downstream automation components generate precise web scraping logic.

### Which LLM does Spider Creator use for generating Mermaid diagrams?

Spider Creator utilizes the `o3_llm` LangChain chat model, which is instantiated in [`shared.py`](https://github.com/carlosplanchon/spidercreator/blob/main/shared.py) and shared across pipeline components. This model is specifically wrapped with a Pydantic `Mermaid` schema in [`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py) to enforce structured output and ensure valid Mermaid syntax generation.

### How does Spider Creator ensure valid Mermaid syntax output?

The architecture enforces validity through **structured generation** using Pydantic models. The `make_mermaid_mindmap` function binds the LLM to a `Mermaid` schema that requires a `mermaid_code` string field. Additionally, the `MERMAID_WORK_MINDMAP_PROMPT` provides detailed syntax rules, step numbering conventions, and few-shot examples to guide the LLM toward syntactically correct output.

### Can I customize the mindmap prompt for specific browsing workflows?

Yes, you can modify the `MERMAID_WORK_MINDMAP_PROMPT` constant defined in [`pipeline/mindmap.py`](https://github.com/carlosplanchon/spidercreator/blob/main/pipeline/mindmap.py) (lines 21-66) to customize the mindmap structure, node hierarchy, or action categorization. When adjusting the prompt, ensure you maintain the Pydantic schema constraints defined in the `Mermaid` class to prevent output validation errors during LLM invocation.