How to Handle API Key Errors and Authentication Failures in SpiderFoot Modules

SpiderFoot modules use a standardized error-handling pattern: verify the API key in handleEvent(), set self.errorState = True on any authentication failure, and abort further processing to prevent redundant requests.

Every SpiderFoot module inherits from SpiderFootPlugin and implements the same defensive checks for missing or invalid credentials. This guide walks through the exact implementation patterns found in the smicallef/spiderfoot source code, including concrete examples from production modules like sfp_virustotal and sfp_xforce.

The Core Architecture: errorState and Early Returns

SpiderFoot's plugin system revolves around a simple boolean flag: errorState. Defined in the base SpiderFootPlugin class ([spiderfoot/plugin.py](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py)), this flag acts as a circuit breaker. Once set to True, the module stops all subsequent API requests for the duration of the scan.

Every handleEvent() method begins with this guard:

def handleEvent(self, event):
    if self.errorState:
        return          # skip processing after a fatal error

This pattern prevents a cascade of failing requests when authentication has already failed.

Detecting Missing API Keys Before Requesting

The first line of defense occurs before any network call. Modules check their opts dictionary—which stores configuration including api_key—and fail fast if credentials are absent.

Pattern: Empty Key Check

In [modules/sfp_virustotal.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L184-L186), lines 184-186 implement this check:

if self.opts["api_key"] == "":
    self.error(
        f"You enabled {self.__class__.__name__} but did not set an API key!"
    )
    self.errorState = True
    return

The [sfp_xforce.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_xforce.py#L215-L216) module extends this pattern for multi-field authentication, checking both api_key and api_password at lines 215-216:

if self.opts['api_key'] == "" or self.opts['api_password'] == "":
    self.error("You enabled sfp_xforce but did not set an API key/password!")
    return

Reusable Implementation Template

def handleEvent(self, event):
    # Circuit breaker

    if self.errorState:
        return

    # Validate required credential

    if not self.opts.get("api_key"):
        self.error(
            f"You enabled {self.__class__.__name__} but did not set an API key!"
        )
        self.errorState = True
        return

    # Proceed with API calls...

Handling Authentication Failures After API Requests

Even with a key present, APIs reject invalid credentials or throttle excessive requests. SpiderFoot modules inspect HTTP response codes and JSON error payloads to detect these conditions.

Throttling: HTTP 204 Handling

VirusTotal rate-limits free tier requests. In [modules/sfp_virustotal.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154), lines 151-154 detect the throttling response:

if res['code'] == "204":
    self.error("You are being rate-limited by VirusTotal.")
    self.errorState = True
    return None

Generic HTTP Authentication Errors

For standard 401/403 responses, modules follow this pattern:

def queryEndpoint(self, query):
    res = self.sf.fetchUrl(
        f"https://api.example.com/search?query={query}",
        headers={"Authorization": f"Bearer {self.opts['api_key']}"},
        timeout=self.opts['_fetchtimeout'],
        useragent=self.opts['_useragent']
    )

    # Authentication failure detection

    if res['code'] in ("401", "403"):
        self.error("Invalid API credentials — authentication failed.")
        self.errorState = True
        return None

    # Parse success or handle other errors...

    try:
        return json.loads(res['content'])
    except json.JSONDecodeError as e:
        self.error(f"Invalid JSON response: {e}")
        self.errorState = True
        return None

JSON Error Payload Parsing

Some APIs return 200 OK with embedded error codes. Modules must parse the response body:

data = json.loads(res['content'])

# Example: API-specific error code

if data.get("code") == "AUTH_FAILURE":
    self.error(f"Authentication failed: {data.get('message')}")
    self.errorState = True
    return None

Complete Working Example

Here's a consolidated module snippet demonstrating all authentication handling patterns:

import json
from spiderfoot import SpiderFootEvent, SpiderFootPlugin


class sfp_example(SpiderFootPlugin):
    meta = {
        'name': "Example Module",
        'options': {
            'api_key': '',
            '_fetchtimeout': 30,
            '_useragent': 'SpiderFoot'
        }
    }

    def setup(self, sfc, userOpts=dict()):
        self.sf = sfc
        self.errorState = False
        self.__dataSource__ = "Example API"
        self.results = self.tempStorage()
        
        for opt in userOpts:
            self.opts[opt] = userOpts[opt]

    def watchedEvents(self):
        return ["IP_ADDRESS"]

    def producedEvents(self):
        return ["MALICIOUS_IPADDR"]

    def handleEvent(self, event):
        # Error circuit breaker

        if self.errorState:
            return

        eventName = event.eventType
        srcModuleName = event.module
        eventData = event.data

        # Validate API key presence

        if self.opts["api_key"] == "":
            self.error(
                f"You enabled {self.__class__.__name__} but did not set an API key!"
            )
            self.errorState = True
            return

        # Make API request

        res = self.sf.fetchUrl(
            f"https://api.example.com/v1/check?ip={eventData}",
            headers={"X-API-Key": self.opts["api_key"]},
            timeout=self.opts['_fetchtimeout'],
            useragent=self.opts['_useragent']
        )

        if res['content'] is None:
            return

        # Handle HTTP-level authentication failures

        if res['code'] == "401":
            self.error("API key rejected — authentication failed.")
            self.errorState = True
            return

        if res['code'] == "429":
            self.error("Rate limit exceeded — throttling detected.")
            self.errorState = True
            return

        # Parse and validate response

        try:
            data = json.loads(res['content'])
        except json.JSONDecodeError as e:
            self.error(f"Could not parse API response: {e}")
            return

        # Handle application-level errors

        if data.get("status") == "error":
            if data.get("code") == "INVALID_KEY":
                self.error("API reports invalid key.")
                self.errorState = True
            else:
                self.error(f"API error: {data.get('message')}")
            return

        # Process successful response...

        if data.get("malicious"):
            evt = SpiderFootEvent(
                "MALICIOUS_IPADDR",
                f"{eventData} [{data['threat_score']}]",
                self.__name__,
                event
            )
            self.notifyListeners(evt)

Key Source Files and Reference Locations

File	Lines	Purpose
[modules/sfp_virustotal.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L184-L186)	184–186	Missing API key validation
[modules/sfp_virustotal.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154)	151–154	Rate limit (HTTP 204) handling
[modules/sfp_xforce.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_xforce.py#L215-L216)	215–216	Multi-field credential check
[modules/sfp_zonefiles.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_zonefiles.py#L154-L156)	154–156	Empty key guard pattern
[spiderfoot/plugin.py](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py)	—	Base SpiderFootPlugin class with errorState definition

Summary

• Check credentials first — Validate api_key and related fields in handleEvent() before any network request.
• Use errorState as a circuit breaker — Set self.errorState = True on any authentication or fatal error, then return immediately.
• Inspect HTTP codes and response bodies — Handle 401/403 for auth failures, 429/204 for throttling, and parse JSON error payloads.
• Fail fast with clear messages — Call self.error() with actionable descriptions so users understand configuration problems.
• Prevent redundant requests — The errorState pattern ensures modules stop querying after the first failure, preserving API quotas and scan performance.

Frequently Asked Questions

How does SpiderFoot prevent modules from hammering APIs after an authentication failure?

Each module inherits errorState from SpiderFootPlugin. When set to True, the initial guard clause in handleEvent() returns immediately, skipping all further event processing for that module during the scan. This single flag prevents redundant network requests and protects API rate limits.

What HTTP status codes do SpiderFoot modules typically check for authentication errors?

Common checks include 401 (Unauthorized) and 403 (Forbidden) for invalid credentials, plus 429 (Too Many Requests) and 204 (No Content, used by VirusTotal) for throttling. Modules like [sfp_virustotal.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154) demonstrate these patterns explicitly.

Why do some modules check for API keys in handleEvent() rather than setup()?

SpiderFoot calls setup() once per module initialization, but users may change options during configuration. Checking in handleEvent() ensures the latest opts values are validated immediately before use. This runtime check catches configuration errors that would otherwise cause silent failures.

Can I customize error handling for a third-party API with unique error formats?

Yes. Override handleEvent() or create a dedicated query method that parses your API's specific error responses—whether JSON fields, HTTP headers, or status codes. Follow the established pattern: log via self.error(), set self.errorState = True, and return early to halt further processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}