How SpiderFoot Modules Communicate Using the Pub/Sub Event System

SpiderFoot modules communicate through a lightweight publish‑subscribe architecture where modules declare consumes and provides event types, the core engine publishes events via notifyListeners, and matching modules receive them through their handleEvent method.

SpiderFoot's modular design enables over 250 reconnaissance plugins to share intelligence without tight coupling. Understanding how SpiderFoot modules communicate using the pub/sub event system is essential for developing custom plugins or debugging scan workflows. This article breaks down the implementation in the smicallef/spiderfoot repository.

How the Pub/Sub Event System Works

SpiderFoot's event system follows five distinct stages that repeat until a scan completes.

Module Registration and Event Declaration

At startup, sf.py loads every module and stores its metadata in the core's module registry. Each module defines two critical lists:

• provides — the event types this module generates
• consumes — the event types this module wants to receive

These declarations live in the module class definition, inherited from SpiderFootPlugin in modules/__init__.py.

# From modules/__init__.py - base class structure

class SpiderFootPlugin:
    # Metadata that every module overrides

    provides = []   # Event types this module produces

    consumes = []   # Event types this module wants to handle

    categories = [] # Classification for UI organization

    
    def handleEvent(self, event):
        """Override this to process consumed events."""
        pass

Real modules like sfp_dnsresolve.py implement these lists concretely:

# modules/sfp_dnsresolve.py (simplified)

class sfp_dnsresolve(SpiderFootPlugin):
    consumes = ['DOMAIN_NAME', 'INTERNET_NAME']
    provides = ['IP_ADDRESS', 'HOSTNAME']

Event Creation with SpiderFootEvent

When a module discovers data, it packages that information into a SpiderFootEvent object. The spiderfoot/event.py file implements this data carrier.

# spiderfoot/event.py - SpiderFootEvent constructor

class SpiderFootEvent:
    def __init__(self, eventType, data, module, sourceEvent, 
                 confidence=100, visibility=100, risk=0):
        self.eventType = eventType      # e.g., 'IP_ADDRESS', 'DOMAIN_NAME'

        self.data = data                # The actual discovered data

        self.module = module            # Which module created this

        self.sourceEvent = sourceEvent  # Parent event in the chain

        self.confidence = confidence    # 0-100 reliability score

The eventType string is the pub/sub routing key. Common types include DOMAIN_NAME, IP_ADDRESS, EMAIL_ADDRESS, URL, and MX_RECORD.

Publishing Events via notifyListeners

The core engine in sf.py implements the central publishing mechanism through notifyListeners. This method is the heart of how SpiderFoot modules communicate using the pub/sub event system.

# sf.py - simplified notifyListeners implementation

def notifyListeners(self, event):
    """Publish an event to all subscribed modules."""
    
    # Iterate every registered module

    for modName, module in self.moduleInstances.items():
        
        # Check if this module wants this event type

        if event.eventType in module.consumes or '*' in module.consumes:
            
            # Invoke the subscriber's handler

            module.handleEvent(event)
            
            # Collect any events the module returns

            returnedEvents = module.handleEvent(event)
            if returnedEvents:
                for newEvent in returnedEvents:
                    self.eventQueue.append(newEvent)

The wildcard * in consumes allows modules to receive every event type, useful for logging or correlation modules.

Subscription Matching and Dispatch

SpiderFoot's dispatch logic performs simple string matching between the published eventType and each module's consumes list. No complex topic hierarchies or pattern matching exists—exact matches or wildcards only.

# Event routing example from sf.py

def dispatchEvent(self, event):
    """Route event to appropriate handlers."""
    
    # Check each module's subscription

    for module in self.loadedModules.values():
        should_receive = (
            event.eventType in module.consumes or
            '*' in module.consumes
        )
        
        if should_receive:
            # Synchronous call to handler

            self.currentModule = module
            returned = module.handleEvent(event)
            self.processReturnedEvents(returned)

This design prioritizes simplicity and traceability over advanced routing features.

Event Chaining and Scan Termination

Events form a parent-child chain through the sourceEvent parameter, enabling data lineage tracking. When sfp_dnsresolve.py receives a DOMAIN_NAME and produces an IP_ADDRESS, the new event references its origin:

# modules/sfp_dnsresolve.py - event chaining

def handleEvent(self, event):
    if event.eventType == 'DOMAIN_NAME':
        domain = event.data
        
        # Perform DNS resolution...

        for ip in resolved_ips:
            # Create child event linked to parent

            ip_event = SpiderFootEvent(
                'IP_ADDRESS', 
                ip, 
                self,           # This module

                event           # Parent event (the DOMAIN_NAME)

            )
            self.notifyListeners(ip_event)  # Publish to next subscribers

The scan terminates when notifyListeners processes all events and the queue empties—no module generates further events.

Practical Implementation: Building a Custom Module

Here's a complete custom module demonstrating the pub/sub pattern:

# modules/sfp_geoiplookup.py

from spiderfoot import SpiderFootEvent, SpiderFootPlugin

class sfp_geoiplookup(SpiderFootPlugin):
    """
    Example module showing pub/sub event flow.
    Consumes IP addresses, produces geolocation events.
    """
    
    # Declare pub/sub interface

    consumes = ['IP_ADDRESS']
    provides = ['GEOLOCATION_INFO', 'COUNTRY_NAME']
    
    def setup(self, sf, userOpts=None):
        self.sf = sf
        self.results = {}
        
    def watchedEvents(self):
        """Required: return consumable event types."""
        return self.consumes
    
    def producedEvents(self):
        """Required: return producible event types."""
        return self.provides
    
    def handleEvent(self, event):
        """Receive IP_ADDRESS, emit geolocation data."""
        
        # Prevent duplicate processing

        ip = event.data
        if ip in self.results:
            return None
        self.results[ip] = True
        
        # Lookup geolocation (simplified)

        geo_data = self.query_geoip_service(ip)
        
        if geo_data:
            # Publish COUNTRY_NAME event

            country_evt = SpiderFootEvent(
                'COUNTRY_NAME',
                geo_data['country'],
                self,
                event
            )
            self.notifyListeners(country_evt)
            
            # Publish GEOLOCATION_INFO event

            geo_evt = SpiderFootEvent(
                'GEOLOCATION_INFO',
                f"{geo_data['city']}, {geo_data['region']}",
                self,
                event
            )
            self.notifyListeners(geo_evt)
            
        return None
    
    def query_geoip_service(self, ip):
        """Placeholder for actual GeoIP lookup."""
        return {'country': 'United States', 'city': 'San Francisco', 'region': 'CA'}

Key implementation requirements:

• Inherit from SpiderFootPlugin in modules/__init__.py
• Override handleEvent to process incoming events
• Use self.notifyListeners() to publish new events
• Always pass self and the source event to SpiderFootEvent constructors

Core Files and Their Pub/Sub Responsibilities

File	Pub/Sub Role	Key Components
sf.py	Central broker	notifyListeners(), module registry, event queue management
spiderfoot/event.py	Event data structure	SpiderFootEvent class with type, data, source chaining
modules/__init__.py	Module base contract	SpiderFootPlugin base class, consumes/provides metadata
modules/sfp_dnsresolve.py	Reference implementation	Demonstrates DOMAIN_NAME → IP_ADDRESS event transformation
test/unit/spiderfoot/test_spiderfootplugin.py	Behavior verification	Unit tests validating event propagation between modules

Performance and Scalability Characteristics

SpiderFoot's pub/sub implementation has specific trade-offs:

• Synchronous dispatch: handleEvent calls block; slow modules delay the entire pipeline
• In-memory queue: Events queue in Python lists during notifyListeners iteration
• No persistence: Event loss occurs if the process crashes mid-scan
• Single-threaded per scan: Modules don't run in parallel for a single target

These constraints keep the architecture simple but require care when handling slow network operations. Modules should delegate blocking work to threads or use self.sf.fetchUrl() helpers that implement internal caching.

Summary

• SpiderFoot modules communicate using the pub/sub event system through consumes and provides metadata declarations in modules/__init__.py
• sf.py implements the central broker with notifyListeners() dispatching events to matching subscribers
• SpiderFootEvent in spiderfoot/event.py carries typed data with parent-child lineage tracking
• Module handlers in handleEvent() receive events synchronously and publish new events via return values or notifyListeners()
• Scans terminate naturally when the event queue empties—no explicit stop signal required
• Over 250 modules interoperate through this decoupled design without direct imports or dependencies

Frequently Asked Questions

How does SpiderFoot prevent infinite event loops between modules?

SpiderFoot tracks event uniqueness through the SpiderFootEvent hash and module-specific results dictionaries. Modules typically check if event.data in self.results before processing, as shown in sfp_dnsresolve.py. The core also limits total events per scan via the sf.py configuration.

Can a module subscribe to multiple event types?

Yes. The consumes list accepts any number of event type strings. For example, sfp_dnsresolve.py uses consumes = ['DOMAIN_NAME', 'INTERNET_NAME'] to handle both direct domains and discovered hostnames. Wildcard subscription with consumes = ['*'] receives every event type.

What happens if two modules both provide the same event type?

Both modules execute independently when their respective consumes criteria match. The core in sf.py iterates through all registered modules for every event—there's no exclusivity. Duplicate data detection happens downstream through the database layer, not the pub/sub system itself.

How do I debug why my module isn't receiving events?

Verify three things in sf.py logs: (1) your module loaded successfully during startup, (2) the consumes list contains the exact event type string (case-sensitive), and (3) no earlier module raised an unhandled exception that halted dispatch. Enable debug mode in SpiderFoot settings to trace notifyListeners calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}