# How SpiderFoot Modules Communicate Using the Pub/Sub Event System

> Discover how SpiderFoot modules communicate via a pub/sub system. Learn about consumes, provides, notifyListeners, and handleEvent for efficient data exchange.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: internals
- Published: 2026-08-15

---

**SpiderFoot modules communicate through a lightweight publish‑subscribe architecture where modules declare `consumes` and `provides` event types, the core engine publishes events via `notifyListeners`, and matching modules receive them through their `handleEvent` method.**

SpiderFoot's modular design enables over 250 reconnaissance plugins to share intelligence without tight coupling. Understanding how SpiderFoot modules communicate using the pub/sub event system is essential for developing custom plugins or debugging scan workflows. This article breaks down the implementation in the smicallef/spiderfoot repository.

## How the Pub/Sub Event System Works

SpiderFoot's event system follows five distinct stages that repeat until a scan completes.

### Module Registration and Event Declaration

At startup, [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) loads every module and stores its metadata in the core's module registry. Each module defines two critical lists:

- **`provides`** — the event types this module generates
- **`consumes`** — the event types this module wants to receive

These declarations live in the module class definition, inherited from `SpiderFootPlugin` in [`modules/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/__init__.py).

```python

# From modules/__init__.py - base class structure

class SpiderFootPlugin:
    # Metadata that every module overrides

    provides = []   # Event types this module produces

    consumes = []   # Event types this module wants to handle

    categories = [] # Classification for UI organization

    
    def handleEvent(self, event):
        """Override this to process consumed events."""
        pass

```

Real modules like [`sfp_dnsresolve.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_dnsresolve.py) implement these lists concretely:

```python

# modules/sfp_dnsresolve.py (simplified)

class sfp_dnsresolve(SpiderFootPlugin):
    consumes = ['DOMAIN_NAME', 'INTERNET_NAME']
    provides = ['IP_ADDRESS', 'HOSTNAME']

```

### Event Creation with SpiderFootEvent

When a module discovers data, it packages that information into a `SpiderFootEvent` object. The [`spiderfoot/event.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/event.py) file implements this data carrier.

```python

# spiderfoot/event.py - SpiderFootEvent constructor

class SpiderFootEvent:
    def __init__(self, eventType, data, module, sourceEvent, 
                 confidence=100, visibility=100, risk=0):
        self.eventType = eventType      # e.g., 'IP_ADDRESS', 'DOMAIN_NAME'

        self.data = data                # The actual discovered data

        self.module = module            # Which module created this

        self.sourceEvent = sourceEvent  # Parent event in the chain

        self.confidence = confidence    # 0-100 reliability score

```

The `eventType` string is the pub/sub routing key. Common types include `DOMAIN_NAME`, `IP_ADDRESS`, `EMAIL_ADDRESS`, `URL`, and `MX_RECORD`.

### Publishing Events via notifyListeners

The core engine in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) implements the central publishing mechanism through `notifyListeners`. This method is the heart of how SpiderFoot modules communicate using the pub/sub event system.

```python

# sf.py - simplified notifyListeners implementation

def notifyListeners(self, event):
    """Publish an event to all subscribed modules."""
    
    # Iterate every registered module

    for modName, module in self.moduleInstances.items():
        
        # Check if this module wants this event type

        if event.eventType in module.consumes or '*' in module.consumes:
            
            # Invoke the subscriber's handler

            module.handleEvent(event)
            
            # Collect any events the module returns

            returnedEvents = module.handleEvent(event)
            if returnedEvents:
                for newEvent in returnedEvents:
                    self.eventQueue.append(newEvent)

```

The wildcard `*` in `consumes` allows modules to receive every event type, useful for logging or correlation modules.

### Subscription Matching and Dispatch

SpiderFoot's dispatch logic performs simple string matching between the published `eventType` and each module's `consumes` list. No complex topic hierarchies or pattern matching exists—exact matches or wildcards only.

```python

# Event routing example from sf.py

def dispatchEvent(self, event):
    """Route event to appropriate handlers."""
    
    # Check each module's subscription

    for module in self.loadedModules.values():
        should_receive = (
            event.eventType in module.consumes or
            '*' in module.consumes
        )
        
        if should_receive:
            # Synchronous call to handler

            self.currentModule = module
            returned = module.handleEvent(event)
            self.processReturnedEvents(returned)

```

This design prioritizes simplicity and traceability over advanced routing features.

### Event Chaining and Scan Termination

Events form a parent-child chain through the `sourceEvent` parameter, enabling data lineage tracking. When [`sfp_dnsresolve.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_dnsresolve.py) receives a `DOMAIN_NAME` and produces an `IP_ADDRESS`, the new event references its origin:

```python

# modules/sfp_dnsresolve.py - event chaining

def handleEvent(self, event):
    if event.eventType == 'DOMAIN_NAME':
        domain = event.data
        
        # Perform DNS resolution...

        for ip in resolved_ips:
            # Create child event linked to parent

            ip_event = SpiderFootEvent(
                'IP_ADDRESS', 
                ip, 
                self,           # This module

                event           # Parent event (the DOMAIN_NAME)

            )
            self.notifyListeners(ip_event)  # Publish to next subscribers

```

The scan terminates when `notifyListeners` processes all events and the queue empties—no module generates further events.

## Practical Implementation: Building a Custom Module

Here's a complete custom module demonstrating the pub/sub pattern:

```python

# modules/sfp_geoiplookup.py

from spiderfoot import SpiderFootEvent, SpiderFootPlugin

class sfp_geoiplookup(SpiderFootPlugin):
    """
    Example module showing pub/sub event flow.
    Consumes IP addresses, produces geolocation events.
    """
    
    # Declare pub/sub interface

    consumes = ['IP_ADDRESS']
    provides = ['GEOLOCATION_INFO', 'COUNTRY_NAME']
    
    def setup(self, sf, userOpts=None):
        self.sf = sf
        self.results = {}
        
    def watchedEvents(self):
        """Required: return consumable event types."""
        return self.consumes
    
    def producedEvents(self):
        """Required: return producible event types."""
        return self.provides
    
    def handleEvent(self, event):
        """Receive IP_ADDRESS, emit geolocation data."""
        
        # Prevent duplicate processing

        ip = event.data
        if ip in self.results:
            return None
        self.results[ip] = True
        
        # Lookup geolocation (simplified)

        geo_data = self.query_geoip_service(ip)
        
        if geo_data:
            # Publish COUNTRY_NAME event

            country_evt = SpiderFootEvent(
                'COUNTRY_NAME',
                geo_data['country'],
                self,
                event
            )
            self.notifyListeners(country_evt)
            
            # Publish GEOLOCATION_INFO event

            geo_evt = SpiderFootEvent(
                'GEOLOCATION_INFO',
                f"{geo_data['city']}, {geo_data['region']}",
                self,
                event
            )
            self.notifyListeners(geo_evt)
            
        return None
    
    def query_geoip_service(self, ip):
        """Placeholder for actual GeoIP lookup."""
        return {'country': 'United States', 'city': 'San Francisco', 'region': 'CA'}

```

Key implementation requirements:

- Inherit from `SpiderFootPlugin` in [`modules/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/__init__.py)
- Override `handleEvent` to process incoming events
- Use `self.notifyListeners()` to publish new events
- Always pass `self` and the source `event` to `SpiderFootEvent` constructors

## Core Files and Their Pub/Sub Responsibilities

| File | Pub/Sub Role | Key Components |
|------|-----------|----------------|
| [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) | Central broker | `notifyListeners()`, module registry, event queue management |
| [`spiderfoot/event.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/event.py) | Event data structure | `SpiderFootEvent` class with type, data, source chaining |
| [`modules/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/__init__.py) | Module base contract | `SpiderFootPlugin` base class, `consumes`/`provides` metadata |
| [`modules/sfp_dnsresolve.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_dnsresolve.py) | Reference implementation | Demonstrates DOMAIN_NAME → IP_ADDRESS event transformation |
| [`test/unit/spiderfoot/test_spiderfootplugin.py`](https://github.com/smicallef/spiderfoot/blob/main/test/unit/spiderfoot/test_spiderfootplugin.py) | Behavior verification | Unit tests validating event propagation between modules |

## Performance and Scalability Characteristics

SpiderFoot's pub/sub implementation has specific trade-offs:

- **Synchronous dispatch**: `handleEvent` calls block; slow modules delay the entire pipeline
- **In-memory queue**: Events queue in Python lists during `notifyListeners` iteration
- **No persistence**: Event loss occurs if the process crashes mid-scan
- **Single-threaded per scan**: Modules don't run in parallel for a single target

These constraints keep the architecture simple but require care when handling slow network operations. Modules should delegate blocking work to threads or use `self.sf.fetchUrl()` helpers that implement internal caching.

## Summary

- SpiderFoot modules communicate using the pub/sub event system through `consumes` and `provides` metadata declarations in [`modules/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/__init__.py)
- [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) implements the central broker with `notifyListeners()` dispatching events to matching subscribers
- `SpiderFootEvent` in [`spiderfoot/event.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/event.py) carries typed data with parent-child lineage tracking
- Module handlers in `handleEvent()` receive events synchronously and publish new events via return values or `notifyListeners()`
- Scans terminate naturally when the event queue empties—no explicit stop signal required
- Over 250 modules interoperate through this decoupled design without direct imports or dependencies

## Frequently Asked Questions

### How does SpiderFoot prevent infinite event loops between modules?

SpiderFoot tracks event uniqueness through the `SpiderFootEvent` hash and module-specific `results` dictionaries. Modules typically check `if event.data in self.results` before processing, as shown in [`sfp_dnsresolve.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_dnsresolve.py). The core also limits total events per scan via the [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) configuration.

### Can a module subscribe to multiple event types?

Yes. The `consumes` list accepts any number of event type strings. For example, [`sfp_dnsresolve.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_dnsresolve.py) uses `consumes = ['DOMAIN_NAME', 'INTERNET_NAME']` to handle both direct domains and discovered hostnames. Wildcard subscription with `consumes = ['*']` receives every event type.

### What happens if two modules both provide the same event type?

Both modules execute independently when their respective `consumes` criteria match. The core in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) iterates through all registered modules for every event—there's no exclusivity. Duplicate data detection happens downstream through the database layer, not the pub/sub system itself.

### How do I debug why my module isn't receiving events?

Verify three things in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) logs: (1) your module loaded successfully during startup, (2) the `consumes` list contains the exact event type string (case-sensitive), and (3) no earlier module raised an unhandled exception that halted dispatch. Enable `debug` mode in SpiderFoot settings to trace `notifyListeners` calls.