# How to Integrate External Tools Like Nmap, DNSTwist, and WhatWeb with SpiderFoot > Learn how to integrate external tools like Nmap, DNSTwist, and WhatWeb with SpiderFoot. Enhance your reconnaissance with SpiderFoot's powerful plugin architecture. - Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot) - Tags: how-to-guide - Published: 2026-08-15 --- **SpiderFoot integrates external reconnaissance tools through a modular plugin architecture where each binary is wrapped in a tool plugin that consumes events, executes commands via subprocess, parses output, and emits enriched intelligence events.** SpiderFoot's extensible design allows security researchers to augment OSINT capabilities by plugging in command-line utilities. The framework handles tool execution, output parsing, and event routing automatically—turning raw command output into structured data within SpiderFoot's intelligence graph. This guide examines how the Nmap, DNSTwist, and WhatWeb integrations work at the source code level. ## How SpiderFoot Tool Plugins Work SpiderFoot uses a plugin-based architecture located in `modules/`. Each external tool requires a dedicated wrapper plugin inheriting from `SpiderFootPlugin` that bridges SpiderFoot's event system with the external binary. ### Core Plugin Responsibilities Every tool plugin performs four essential functions: - **Event subscription** — declares which SpiderFoot events trigger execution via `watchedEvents()` - **Configuration management** — stores binary paths and tool-specific options in `self.opts` - **Execution handling** — launches the tool via `subprocess.Popen` with timeouts and error handling - **Output transformation** — parses tool output and creates new SpiderFoot events via `SpiderFootEvent()` ## Nmap Integration Deep Dive The Nmap plugin resides in [`modules/sfp_tool_nmap.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_tool_nmap.py) and provides operating system fingerprinting capabilities. ### Configuration and Path Validation The plugin validates the Nmap binary path during setup: ```python # From modules/sfp_tool_nmap.py - path validation logic if not os.path.isfile(self.opts['toolnmap_path']): self.error(f"File does not exist: {self.opts['toolnmap_path']}") self.errorState = True return ``` Options are defined in `opts` and `optdescs`, including `toolnmap_path` (binary location), aggression level, and scan timing. ### Input Sanitization and Execution Before invocation, targets are validated using SpiderFoot's helper utilities: ```python # From modules/sfp_tool_nmap.py - target validation target = event.data if not SpiderFootHelpers.validIP(target) and not SpiderFootHelpers.sanitiseInput(target): self.error(f"Invalid target: {target}") return ``` The plugin constructs command-line arguments and executes with timeout protection: ```python # Command construction and execution pattern cmd = [ self.opts['toolnmap_path'], "-O", # OS detection "--osscan-limit", "-T4", target ] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate(timeout=300) ``` ### Parsing Nmap Output Nmap's plain-text output is parsed line-by-line to extract OS details: ```python # Output parsing for OS fingerprinting for line in stdout.decode('utf-8').split("\n"): if "OS details:" in line or "Running:" in line: os_info = line.split(":", 1)[1].strip() evt = SpiderFootEvent("OPERATING_SYSTEM", os_info, self.__name__, event) self.notifyListeners(evt) ``` ## WhatWeb Integration Deep Dive The WhatWeb plugin in [`modules/sfp_tool_whatweb.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_tool_whatweb.py) performs web technology fingerprinting using Ruby's WhatWeb scanner. ### Ruby Environment Setup WhatWeb requires Ruby execution, so the plugin manages both Ruby and WhatWeb paths: ```python # From modules/sfp_tool_whatweb.py - dual path validation if not os.path.isfile(self.opts['toolwhatweb_path']): self.error(f"WhatWeb binary not found: {self.opts['toolwhatweb_path']}") self.errorState = True return if not os.path.isfile(self.opts['toolwhatweb_ruby_path']): self.error(f"Ruby binary not found: {self.opts['toolwhatweb_ruby_path']}") self.errorState = True return ``` ### JSON Output Processing WhatWeb produces structured JSON output via the `--log-json` flag, simplifying parsing: ```python # From modules/sfp_tool_whatweb.py - JSON parsing cmd = [ self.opts['toolwhatweb_ruby_path'], self.opts['toolwhatweb_path'], f"--aggression={self.opts['toolwhatweb_aggression']}", "--log-json", target ] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = p.communicate(timeout=300) try: results = json.loads(stdout.decode('utf-8')) for result in results: if 'plugins' in result: for plugin_name, plugin_data in result['plugins'].items(): # Convert to SpiderFoot events evt = SpiderFootEvent("WEBSERVER_TECHNOLOGY", f"{plugin_name}: {plugin_data}", self.__name__, event) self.notifyListeners(evt) except json.JSONDecodeError: self.error("Failed to parse WhatWeb JSON output") ``` ### Error State Management The plugin sets `self.errorState = True` on execution failures to prevent repeated attempts during a scan. ## DNSTwist Integration Pattern DNSTwist follows the identical plugin pattern in [`modules/sfp_tool_dnstwist.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_tool_dnstwist.py), specializing in domain generation algorithm (DGA) and typosquatting detection. ### Domain-Focused Input Handling DNSTwist operates on domain names rather than IP addresses, consuming `INTERNET_NAME` and `DOMAIN_NAME` events. ### JSON Output for Domain Variations Like WhatWeb, DNSTwist provides JSON output that the plugin iterates to extract domain permutations: ```python # Typical DNSTwist execution pattern cmd = [ self.opts['dnstwist_path'], "-j", # JSON output "-r", # perform DNS resolution domain ] # Parse permutations and generate SIMILARDOMAIN events results = json.loads(stdout.decode('utf-8')) for entry in results: if 'domain' in entry and entry['domain'] != domain: # Create event for potential typosquat evt = SpiderFootEvent("SIMILARDOMAIN", entry['domain'], self.__name__, event) self.notifyListeners(evt) ``` ## Running External Tools in SpiderFoot Scans ### Programmatic Execution Example ```python from spiderfoot import SpiderFoot, SpiderFootEvent # Initialize SpiderFoot instance sf = SpiderFoot() sf.setConfig({ # Nmap configuration "toolnmap_path": "/usr/bin/nmap", "toolnmap_aggression": "5", # WhatWeb configuration "toolwhatweb_path": "/usr/local/bin/whatweb", "toolwhatweb_ruby_path": "/usr/bin/ruby", "toolwhatweb_aggression": "3", # DNSTwist configuration "tooldnstwist_path": "/usr/local/bin/dnstwist" }) sf.start() # Trigger WhatWeb scan whatweb_event = SpiderFootEvent("INTERNET_NAME", "target.example.com", "manual_trigger") sf.processEvent(whatweb_event) # Trigger Nmap scan nmap_event = SpiderFootEvent("IP_ADDRESS", "93.184.216.34", "manual_trigger") sf.processEvent(nmap_event) ``` ### Configuration via Web Interface Tool paths and aggression levels are configurable through SpiderFoot's web UI under module settings. The plugins expose these via `optdescs` dictionaries that render as form fields. ## Security and Reliability Considerations ### Input Sanitization Both [`sfp_tool_nmap.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_tool_nmap.py) and [`sfp_tool_whatweb.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_tool_whatweb.py) leverage `SpiderFootHelpers.sanitiseInput()` to prevent command injection: ```python # Defensive pattern used across tool plugins if not self.sf.validIP(target) and not SpiderFootHelpers.sanitiseInput(target): self.debug(f"Skipping invalid target: {target}") return ``` ### Execution Timeouts All external tools are executed with 300-second timeouts to prevent hung processes from blocking scans indefinitely. ### Error State Isolation Failed plugin initialization sets `self.errorState = True`, ensuring the plugin skips subsequent events rather than repeatedly failing. ## Creating Your Own Tool Plugin Follow this five-step pattern to integrate additional command-line tools: 1. **Inherit from SpiderFootPlugin** — create [`modules/sfp_tool_yourtool.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_tool_yourtool.py) 2. **Define metadata** in the `meta` dictionary: ```python meta = { 'name': "Tool - YourTool", 'summary': "Description of what the tool does", 'toolDetails': { 'name': 'YourTool', 'description': 'Official tool description', 'website': 'https://example.com/yourtool', 'repository': 'https://github.com/user/yourtool' } } ``` 3. **Specify configuration options** in `opts` and `optdescs` 4. **Implement event methods**: - `watchedEvents()` — return list like `["IP_ADDRESS", "INTERNET_NAME"]` - `producedEvents()` — return list like `["RAW_RIR_DATA", "TECHNOLOGY"]` - `handleEvent(self, event)` — core execution logic 5. **Parse and emit** — transform tool output into SpiderFoot events via `SpiderFootEvent()` and `self.notifyListeners(evt)` ## Summary - **SpiderFoot's plugin architecture** in `modules/` wraps external binaries with consistent interfaces for path validation, input sanitization, and event emission - **Nmap integration** ([`sfp_tool_nmap.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_tool_nmap.py)) performs OS fingerprinting by parsing plain-text output for "OS details" lines - **WhatWeb integration** ([`sfp_tool_whatweb.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_tool_whatweb.py)) requires Ruby execution and parses JSON output to extract `WEBSERVER_TECHNOLOGY` events - **DNSTwist integration** follows identical patterns for domain permutation analysis - **All tool plugins** use `subprocess.Popen` with 300-second timeouts, set `errorState` on failures, and validate inputs via `SpiderFootHelpers` - **New tool integrations** follow a five-step pattern: inherit, define metadata, configure, implement event handlers, and parse output ## Frequently Asked Questions ### What happens if the external tool binary is missing or misconfigured? The plugin validates binary paths in its setup method using `os.path.isfile()`. If the path doesn't exist, the plugin logs an error, sets `self.errorState = True`, and skips all subsequent events. This prevents scan failures from repeated execution attempts. Check SpiderFoot's logs for "File does not exist" messages to diagnose path issues. ### Can I use custom command-line flags with these tools? Yes—modify the `opts` dictionary in your plugin or create a subclass. The aggression level is exposed as a configurable option (e.g., `toolnmap_aggression`, `toolwhatweb_aggression`). For custom flags, edit the command list construction in `handleEvent()` before the `subprocess.Popen` call. ### Does SpiderFoot cache or deduplicate external tool results? SpiderFoot's core engine handles event deduplication based on event type, data value, and source. The tool plugins themselves don't implement caching—they emit events for every matching input. For repeated scans against the same targets, SpiderFoot's database layer prevents duplicate storage of identical events. ### Why does WhatWeb require separate Ruby path configuration? WhatWeb is a Ruby script rather than a compiled binary. The plugin requires explicit Ruby interpreter path configuration because Ruby installations vary across systems (system Ruby, rbenv, rvm, etc.). The plugin constructs commands as `[ruby_path, whatweb_path, ...args]` to ensure correct execution regardless of shebang handling or environment variables.