How to Integrate External Tools Like Nmap, DNSTwist, and WhatWeb with SpiderFoot

SpiderFoot integrates external reconnaissance tools through a modular plugin architecture where each binary is wrapped in a tool plugin that consumes events, executes commands via subprocess, parses output, and emits enriched intelligence events.

SpiderFoot's extensible design allows security researchers to augment OSINT capabilities by plugging in command-line utilities. The framework handles tool execution, output parsing, and event routing automatically—turning raw command output into structured data within SpiderFoot's intelligence graph. This guide examines how the Nmap, DNSTwist, and WhatWeb integrations work at the source code level.

How SpiderFoot Tool Plugins Work

SpiderFoot uses a plugin-based architecture located in modules/. Each external tool requires a dedicated wrapper plugin inheriting from SpiderFootPlugin that bridges SpiderFoot's event system with the external binary.

Core Plugin Responsibilities

Every tool plugin performs four essential functions:

• Event subscription — declares which SpiderFoot events trigger execution via watchedEvents()
• Configuration management — stores binary paths and tool-specific options in self.opts
• Execution handling — launches the tool via subprocess.Popen with timeouts and error handling
• Output transformation — parses tool output and creates new SpiderFoot events via SpiderFootEvent()

Nmap Integration Deep Dive

The Nmap plugin resides in modules/sfp_tool_nmap.py and provides operating system fingerprinting capabilities.

Configuration and Path Validation

The plugin validates the Nmap binary path during setup:

# From modules/sfp_tool_nmap.py - path validation logic

if not os.path.isfile(self.opts['toolnmap_path']):
    self.error(f"File does not exist: {self.opts['toolnmap_path']}")
    self.errorState = True
    return

Options are defined in opts and optdescs, including toolnmap_path (binary location), aggression level, and scan timing.

Input Sanitization and Execution

Before invocation, targets are validated using SpiderFoot's helper utilities:

# From modules/sfp_tool_nmap.py - target validation

target = event.data
if not SpiderFootHelpers.validIP(target) and not SpiderFootHelpers.sanitiseInput(target):
    self.error(f"Invalid target: {target}")
    return

The plugin constructs command-line arguments and executes with timeout protection:

# Command construction and execution pattern

cmd = [
    self.opts['toolnmap_path'],
    "-O",  # OS detection

    "--osscan-limit",
    "-T4",
    target
]

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate(timeout=300)

Parsing Nmap Output

Nmap's plain-text output is parsed line-by-line to extract OS details:

# Output parsing for OS fingerprinting

for line in stdout.decode('utf-8').split("\n"):
    if "OS details:" in line or "Running:" in line:
        os_info = line.split(":", 1)[1].strip()
        evt = SpiderFootEvent("OPERATING_SYSTEM", os_info, self.__name__, event)
        self.notifyListeners(evt)

WhatWeb Integration Deep Dive

The WhatWeb plugin in modules/sfp_tool_whatweb.py performs web technology fingerprinting using Ruby's WhatWeb scanner.

Ruby Environment Setup

WhatWeb requires Ruby execution, so the plugin manages both Ruby and WhatWeb paths:

# From modules/sfp_tool_whatweb.py - dual path validation

if not os.path.isfile(self.opts['toolwhatweb_path']):
    self.error(f"WhatWeb binary not found: {self.opts['toolwhatweb_path']}")
    self.errorState = True
    return

if not os.path.isfile(self.opts['toolwhatweb_ruby_path']):
    self.error(f"Ruby binary not found: {self.opts['toolwhatweb_ruby_path']}")
    self.errorState = True
    return

JSON Output Processing

WhatWeb produces structured JSON output via the --log-json flag, simplifying parsing:

# From modules/sfp_tool_whatweb.py - JSON parsing

cmd = [
    self.opts['toolwhatweb_ruby_path'],
    self.opts['toolwhatweb_path'],
    f"--aggression={self.opts['toolwhatweb_aggression']}",
    "--log-json",
    target
]

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = p.communicate(timeout=300)

try:
    results = json.loads(stdout.decode('utf-8'))
    for result in results:
        if 'plugins' in result:
            for plugin_name, plugin_data in result['plugins'].items():
                # Convert to SpiderFoot events

                evt = SpiderFootEvent("WEBSERVER_TECHNOLOGY", 
                                      f"{plugin_name}: {plugin_data}", 
                                      self.__name__, event)
                self.notifyListeners(evt)
except json.JSONDecodeError:
    self.error("Failed to parse WhatWeb JSON output")

Error State Management

The plugin sets self.errorState = True on execution failures to prevent repeated attempts during a scan.

DNSTwist Integration Pattern

DNSTwist follows the identical plugin pattern in modules/sfp_tool_dnstwist.py, specializing in domain generation algorithm (DGA) and typosquatting detection.

Domain-Focused Input Handling

DNSTwist operates on domain names rather than IP addresses, consuming INTERNET_NAME and DOMAIN_NAME events.

JSON Output for Domain Variations

Like WhatWeb, DNSTwist provides JSON output that the plugin iterates to extract domain permutations:

# Typical DNSTwist execution pattern

cmd = [
    self.opts['dnstwist_path'],
    "-j",  # JSON output

    "-r",  # perform DNS resolution

    domain
]

# Parse permutations and generate SIMILARDOMAIN events

results = json.loads(stdout.decode('utf-8'))
for entry in results:
    if 'domain' in entry and entry['domain'] != domain:
        # Create event for potential typosquat

        evt = SpiderFootEvent("SIMILARDOMAIN", entry['domain'], 
                              self.__name__, event)
        self.notifyListeners(evt)

Running External Tools in SpiderFoot Scans

Programmatic Execution Example

from spiderfoot import SpiderFoot, SpiderFootEvent

# Initialize SpiderFoot instance

sf = SpiderFoot()
sf.setConfig({
    # Nmap configuration

    "toolnmap_path": "/usr/bin/nmap",
    "toolnmap_aggression": "5",
    
    # WhatWeb configuration  

    "toolwhatweb_path": "/usr/local/bin/whatweb",
    "toolwhatweb_ruby_path": "/usr/bin/ruby",
    "toolwhatweb_aggression": "3",
    
    # DNSTwist configuration

    "tooldnstwist_path": "/usr/local/bin/dnstwist"
})

sf.start()

# Trigger WhatWeb scan

whatweb_event = SpiderFootEvent("INTERNET_NAME", "target.example.com", 
                                "manual_trigger")
sf.processEvent(whatweb_event)

# Trigger Nmap scan

nmap_event = SpiderFootEvent("IP_ADDRESS", "93.184.216.34", 
                             "manual_trigger")
sf.processEvent(nmap_event)

Configuration via Web Interface

Tool paths and aggression levels are configurable through SpiderFoot's web UI under module settings. The plugins expose these via optdescs dictionaries that render as form fields.

Security and Reliability Considerations

Input Sanitization

Both sfp_tool_nmap.py and sfp_tool_whatweb.py leverage SpiderFootHelpers.sanitiseInput() to prevent command injection:

# Defensive pattern used across tool plugins

if not self.sf.validIP(target) and not SpiderFootHelpers.sanitiseInput(target):
    self.debug(f"Skipping invalid target: {target}")
    return

Execution Timeouts

All external tools are executed with 300-second timeouts to prevent hung processes from blocking scans indefinitely.

Error State Isolation

Failed plugin initialization sets self.errorState = True, ensuring the plugin skips subsequent events rather than repeatedly failing.

Creating Your Own Tool Plugin

Follow this five-step pattern to integrate additional command-line tools:

1. Inherit from SpiderFootPlugin — create modules/sfp_tool_yourtool.py

2. Define metadata in the meta dictionary:

   meta = {
       'name': "Tool - YourTool",
       'summary': "Description of what the tool does",
       'toolDetails': {
           'name': 'YourTool',
           'description': 'Official tool description',
           'website': 'https://example.com/yourtool',
           'repository': 'https://github.com/user/yourtool'
       }
   }

3. Specify configuration options in opts and optdescs

4. Implement event methods:

   • watchedEvents() — return list like ["IP_ADDRESS", "INTERNET_NAME"]
   • producedEvents() — return list like ["RAW_RIR_DATA", "TECHNOLOGY"]
   • handleEvent(self, event) — core execution logic

5. Parse and emit — transform tool output into SpiderFoot events via SpiderFootEvent() and self.notifyListeners(evt)

Summary

• SpiderFoot's plugin architecture in modules/ wraps external binaries with consistent interfaces for path validation, input sanitization, and event emission
• Nmap integration (sfp_tool_nmap.py) performs OS fingerprinting by parsing plain-text output for "OS details" lines
• WhatWeb integration (sfp_tool_whatweb.py) requires Ruby execution and parses JSON output to extract WEBSERVER_TECHNOLOGY events
• DNSTwist integration follows identical patterns for domain permutation analysis
• All tool plugins use subprocess.Popen with 300-second timeouts, set errorState on failures, and validate inputs via SpiderFootHelpers
• New tool integrations follow a five-step pattern: inherit, define metadata, configure, implement event handlers, and parse output

Frequently Asked Questions

What happens if the external tool binary is missing or misconfigured?

The plugin validates binary paths in its setup method using os.path.isfile(). If the path doesn't exist, the plugin logs an error, sets self.errorState = True, and skips all subsequent events. This prevents scan failures from repeated execution attempts. Check SpiderFoot's logs for "File does not exist" messages to diagnose path issues.

Can I use custom command-line flags with these tools?

Yes—modify the opts dictionary in your plugin or create a subclass. The aggression level is exposed as a configurable option (e.g., toolnmap_aggression, toolwhatweb_aggression). For custom flags, edit the command list construction in handleEvent() before the subprocess.Popen call.

Does SpiderFoot cache or deduplicate external tool results?

SpiderFoot's core engine handles event deduplication based on event type, data value, and source. The tool plugins themselves don't implement caching—they emit events for every matching input. For repeated scans against the same targets, SpiderFoot's database layer prevents duplicate storage of identical events.

Why does WhatWeb require separate Ruby path configuration?

WhatWeb is a Ruby script rather than a compiled binary. The plugin requires explicit Ruby interpreter path configuration because Ruby installations vary across systems (system Ruby, rbenv, rvm, etc.). The plugin constructs commands as [ruby_path, whatweb_path, ...args] to ensure correct execution regardless of shebang handling or environment variables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}