How to Use SpiderFoot's Built-In Data Visualizations: A Complete Guide for Security Analysts

SpiderFoot converts raw scan relationships into interactive D3.js charts using the dataParentChildToTree helper in spiderfoot/helpers.py, exposing them through the "Visualisations" tab in the web UI.

SpiderFoot's reconnaissance platform doesn't just collect OSINT data—it transforms discovered relationships into visual graphs that reveal hidden infrastructure patterns. This guide explains how SpiderFoot's built-in data visualizations work, how to access them through the web interface, and how to reuse the visualization logic in your own tools.

How SpiderFoot Data Visualizations Work

SpiderFoot's visualization pipeline follows a clean separation between data extraction and rendering. Understanding this architecture helps you troubleshoot issues or extend the system.

The Core Conversion Helper

All visualizations rely on SpiderFootHelpers.dataParentChildToTree in spiderfoot/helpers.py (lines 363-391). This static method takes a flat parent-child dictionary and returns a nested JSON structure that D3.js expects:

• Each node has a name string
• Each node has an optional children array (or None for leaves)
• The method validates input and guarantees a single root node

The helper enforces type safety: invalid input raises TypeError or ValueError, preventing malformed data from reaching the front end.

The End-to-End Flow

1. Scan execution stores entity relationships in SpiderFoot's SQLite database
2. Web UI request hits the /visualise endpoint (handled in sfwebui.py)
3. Backend processing queries the database, assembles a parent → [children] mapping
4. Conversion passes the mapping to dataParentChildToTree
5. JSON delivery sends the nested structure to the browser
6. D3 rendering draws the interactive visualization

This pipeline means any SpiderFoot scan—launched via web UI, CLI (sfcli.py), or API—automatically supports visualization once complete.

Built-In Visualization Types

SpiderFoot's web UI exposes three primary chart types through the Visualisations tab. Each serves different investigative purposes.

Domain Graph

What it shows: DNS hierarchy including domains, subdomains, and name servers

Source data: Parent-child relationships from DNS record modules

How it's built:

• dataParentChildToTree creates a D3-compatible tree structure
• Front-end renders with d3-tree layout
• Natural top-down reading matches DNS delegation chains

Use this when tracking domain infrastructure or identifying unexpected subdomains.

Network Graph

What it shows: IP address and host relationships—who hosts what

Source data: IP ↔ hostname mappings stored during scanning

How it's built:

• Same helper generates the JSON payload
• UI applies a force-directed D3 layout instead of a tree
• Physics simulation clusters related hosts naturally

This view excels at identifying shared hosting infrastructure and CDN usage patterns.

Entity Relationship Diagram

What it shows: Multi-hop connections between entities (emails, persons, companies, domains)

Source data: Entity-entity edges from correlation modules

How it's built:

• Helper constructs a tree where each entity becomes a node
• Relationships become parent-child or sibling connections
• D3 renders as an explorable network graph

Use this for social network analysis and supply chain mapping.

Accessing Visualizations Through the Web UI

The simplest way to use SpiderFoot's data visualizations is through the built-in web interface.

Step-by-Step: View Your First Graph

1. Start SpiderFoot (Docker: docker run -p 5001:5001 spiderfoot, or local install)
2. Navigate to http://localhost:5001
3. Create or select a scan and run it to completion
4. Click the "Visualisations" tab in the scan results
5. Select your chart type (Domain Graph, Network Graph, or Entity Relationship)
6. Interact with the rendered D3 diagram—zoom, pan, and click nodes to explore

Behind the scenes, the UI issues a GET request to:

/visualise?type=domaingraph&scan_id=<SCAN_ID>

The type parameter selects the visualization variant; scan_id targets your specific scan data. No additional configuration is required—the backend automatically builds the appropriate relationship map from the database.

Generating Visualization Data in Python

You can reuse SpiderFoot's conversion logic outside the bundled UI. This enables custom dashboards, automated reporting, or integration with other analysis tools.

Stand-Alone Tree Generation

from spiderfoot.helpers import SpiderFootHelpers

# Example: DNS parent-child mapping from your own data source

dns_map = {
    "example.com": ["ns1.example.com", "ns2.example.com"],
    "ns1.example.com": ["192.0.2.1"],
    "ns2.example.com": ["192.0.2.2"],
    "192.0.2.1": None,
    "192.0.2.2": None,
}

# Convert to D3-compatible nested structure

tree = SpiderFootHelpers.dataParentChildToTree(dns_map)
print(tree)

Output structure:

{
  "name": "example.com",
  "children": [
    {
      "name": "ns1.example.com",
      "children": [
        {"name": "192.0.2.1", "children": null}
      ]
    },
    {
      "name": "ns2.example.com",
      "children": [
        {"name": "192.0.2.2", "children": null}
      ]
    }
  ]
}

The helper handles edge cases automatically:

• Validates that input is a dictionary
• Guarantees exactly one root node
• Recursively nests children to arbitrary depth
• Raises TypeError or ValueError for malformed input

Embedding SpiderFoot Visualizations in Custom Pages

Extract the JSON from SpiderFoot's API and render it with your own D3 configuration.

Minimal Custom Renderer

<!DOCTYPE html>
<html>
<head>
  <script src="https://d3js.org/d3.v5.min.js"></script>
</head>
<body>
  <div id="chart"></div>

  <script>
    fetch(`/visualise?type=domaingraph&scan_id=<SCAN_ID>`)
      .then(response => response.json())
      .then(data => {
        const root = d3.hierarchy(data);
        const treeLayout = d3.tree().size([800, 600]);
        treeLayout(root);

        const svg = d3.select('#chart')
          .append('svg')
          .attr('width', 800)
          .attr('height', 600);

        // Draw links
        svg.selectAll('.link')
          .data(root.links())
          .enter()
          .append('line')
          .classed('link', true)
          .attr('x1', d => d.source.y)
          .attr('y1', d => d.source.x)
          .attr('x2', d => d.target.y)
          .attr('y2', d => d.target.x)
          .attr('stroke', '#ccc');

        // Draw nodes
        const node = svg.selectAll('.node')
          .data(root.descendants())
          .enter()
          .append('g')
          .classed('node', true)
          .attr('transform', d => `translate(${d.y},${d.x})`);

        node.append('circle').attr('r', 4);
        node.append('text')
          .attr('dx', 6)
          .attr('dy', 3)
          .text(d => d.data.name);
      });
  </script>
</body>
</html>

Replace <SCAN_ID> with your actual scan identifier. The fetched JSON format matches exactly what SpiderFoot's built-in UI consumes—you're simply applying custom D3 styling and layout choices.

Key Implementation Files

File	Purpose	Critical Lines
spiderfoot/helpers.py	Core conversion logic	SpiderFootHelpers.dataParentChildToTree at lines 363-391
sfwebui.py	HTTP endpoint and database queries	/visualise route handler
sfcli.py	CLI entry point that populates visualization data	Scan execution flow
templates/visualisations.html	D3.js front-end rendering	Flask-served template (implicit)

Understanding these files lets you trace any visualization issue from UI symptom back to data source.

Summary

• SpiderFoot's data visualizations rely on a single helper method—dataParentChildToTree in spiderfoot/helpers.py—to convert flat relationship data into nested D3-compatible JSON
• Three built-in chart types cover DNS hierarchies (Domain Graph), infrastructure mapping (Network Graph), and multi-entity correlations (Entity Relationship)
• Access via web UI requires only completing a scan and clicking the Visualisations tab; no configuration needed
• Reuse outside SpiderFoot by importing SpiderFootHelpers or fetching /visualise endpoints for custom dashboards
• All validation and tree-building logic is centralized, making the system robust and extensible

Frequently Asked Questions

Does SpiderFoot require external tools to generate visualizations?

No. SpiderFoot bundles D3.js and handles the entire pipeline internally. The dataParentChildToTree helper produces JSON that any D3 installation can render, but SpiderFoot's web UI includes everything needed for interactive charts without additional dependencies.

Can I visualize data from a scan that ran via CLI instead of the web UI?

Yes. SpiderFoot stores all scan results in the same SQLite database regardless of launch method. The /visualise endpoint reads from this database, so CLI-initiated scans appear in the web UI's Visualisations tab as long as you use the same database file.

What happens if my scan data contains cycles or multiple root nodes?

The dataParentChildToTree helper enforces a strict tree structure. It validates input and raises ValueError if the mapping cannot resolve to a single root. In practice, SpiderFoot's data collection typically produces valid hierarchies; cycles in entity relationships are handled by the Entity Relationship diagram's force-directed layout rather than strict tree rendering.

Is there a way to export visualization data for offline analysis?

Yes. Query the /visualise endpoint directly—the returned JSON is self-contained and portable. You can also use the Python helper to generate trees from exported scan data, then feed the result to any D3-compatible tool or save as a standard JSON file for documentation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}