# How to Handle API Key Errors and Authentication Failures in SpiderFoot Modules

> Learn to handle API key errors and authentication failures in SpiderFoot modules. Discover the standardized error-handling pattern to prevent redundant requests and ensure smooth data collection.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: how-to-guide
- Published: 2026-08-15

---

**SpiderFoot modules use a standardized error-handling pattern: verify the API key in `handleEvent()`, set `self.errorState = True` on any authentication failure, and abort further processing to prevent redundant requests.**

Every SpiderFoot module inherits from `SpiderFootPlugin` and implements the same defensive checks for missing or invalid credentials. This guide walks through the exact implementation patterns found in the [smicallef/spiderfoot](https://github.com/smicallef/spiderfoot) source code, including concrete examples from production modules like `sfp_virustotal` and `sfp_xforce`.

## The Core Architecture: errorState and Early Returns

SpiderFoot's plugin system revolves around a simple boolean flag: **errorState**. Defined in the base `SpiderFootPlugin` class ([[`spiderfoot/plugin.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/plugin.py)](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py)), this flag acts as a circuit breaker. Once set to `True`, the module stops all subsequent API requests for the duration of the scan.

Every `handleEvent()` method begins with this guard:

```python
def handleEvent(self, event):
    if self.errorState:
        return          # skip processing after a fatal error

```

This pattern prevents a cascade of failing requests when authentication has already failed.

## Detecting Missing API Keys Before Requesting

The first line of defense occurs before any network call. Modules check their `opts` dictionary—which stores configuration including `api_key`—and fail fast if credentials are absent.

### Pattern: Empty Key Check

In [[`modules/sfp_virustotal.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_virustotal.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L184-L186), lines **184-186** implement this check:

```python
if self.opts["api_key"] == "":
    self.error(
        f"You enabled {self.__class__.__name__} but did not set an API key!"
    )
    self.errorState = True
    return

```

The [[`sfp_xforce.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_xforce.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_xforce.py#L215-L216) module extends this pattern for multi-field authentication, checking both `api_key` and `api_password` at lines **215-216**:

```python
if self.opts['api_key'] == "" or self.opts['api_password'] == "":
    self.error("You enabled sfp_xforce but did not set an API key/password!")
    return

```

### Reusable Implementation Template

```python
def handleEvent(self, event):
    # Circuit breaker

    if self.errorState:
        return

    # Validate required credential

    if not self.opts.get("api_key"):
        self.error(
            f"You enabled {self.__class__.__name__} but did not set an API key!"
        )
        self.errorState = True
        return

    # Proceed with API calls...

```

## Handling Authentication Failures After API Requests

Even with a key present, APIs reject invalid credentials or throttle excessive requests. SpiderFoot modules inspect HTTP response codes and JSON error payloads to detect these conditions.

### Throttling: HTTP 204 Handling

VirusTotal rate-limits free tier requests. In [[`modules/sfp_virustotal.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_virustotal.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154), lines **151-154** detect the throttling response:

```python
if res['code'] == "204":
    self.error("You are being rate-limited by VirusTotal.")
    self.errorState = True
    return None

```

### Generic HTTP Authentication Errors

For standard 401/403 responses, modules follow this pattern:

```python
def queryEndpoint(self, query):
    res = self.sf.fetchUrl(
        f"https://api.example.com/search?query={query}",
        headers={"Authorization": f"Bearer {self.opts['api_key']}"},
        timeout=self.opts['_fetchtimeout'],
        useragent=self.opts['_useragent']
    )

    # Authentication failure detection

    if res['code'] in ("401", "403"):
        self.error("Invalid API credentials — authentication failed.")
        self.errorState = True
        return None

    # Parse success or handle other errors...

    try:
        return json.loads(res['content'])
    except json.JSONDecodeError as e:
        self.error(f"Invalid JSON response: {e}")
        self.errorState = True
        return None

```

### JSON Error Payload Parsing

Some APIs return 200 OK with embedded error codes. Modules must parse the response body:

```python
data = json.loads(res['content'])

# Example: API-specific error code

if data.get("code") == "AUTH_FAILURE":
    self.error(f"Authentication failed: {data.get('message')}")
    self.errorState = True
    return None

```

## Complete Working Example

Here's a consolidated module snippet demonstrating all authentication handling patterns:

```python
import json
from spiderfoot import SpiderFootEvent, SpiderFootPlugin


class sfp_example(SpiderFootPlugin):
    meta = {
        'name': "Example Module",
        'options': {
            'api_key': '',
            '_fetchtimeout': 30,
            '_useragent': 'SpiderFoot'
        }
    }

    def setup(self, sfc, userOpts=dict()):
        self.sf = sfc
        self.errorState = False
        self.__dataSource__ = "Example API"
        self.results = self.tempStorage()
        
        for opt in userOpts:
            self.opts[opt] = userOpts[opt]

    def watchedEvents(self):
        return ["IP_ADDRESS"]

    def producedEvents(self):
        return ["MALICIOUS_IPADDR"]

    def handleEvent(self, event):
        # Error circuit breaker

        if self.errorState:
            return

        eventName = event.eventType
        srcModuleName = event.module
        eventData = event.data

        # Validate API key presence

        if self.opts["api_key"] == "":
            self.error(
                f"You enabled {self.__class__.__name__} but did not set an API key!"
            )
            self.errorState = True
            return

        # Make API request

        res = self.sf.fetchUrl(
            f"https://api.example.com/v1/check?ip={eventData}",
            headers={"X-API-Key": self.opts["api_key"]},
            timeout=self.opts['_fetchtimeout'],
            useragent=self.opts['_useragent']
        )

        if res['content'] is None:
            return

        # Handle HTTP-level authentication failures

        if res['code'] == "401":
            self.error("API key rejected — authentication failed.")
            self.errorState = True
            return

        if res['code'] == "429":
            self.error("Rate limit exceeded — throttling detected.")
            self.errorState = True
            return

        # Parse and validate response

        try:
            data = json.loads(res['content'])
        except json.JSONDecodeError as e:
            self.error(f"Could not parse API response: {e}")
            return

        # Handle application-level errors

        if data.get("status") == "error":
            if data.get("code") == "INVALID_KEY":
                self.error("API reports invalid key.")
                self.errorState = True
            else:
                self.error(f"API error: {data.get('message')}")
            return

        # Process successful response...

        if data.get("malicious"):
            evt = SpiderFootEvent(
                "MALICIOUS_IPADDR",
                f"{eventData} [{data['threat_score']}]",
                self.__name__,
                event
            )
            self.notifyListeners(evt)

```

## Key Source Files and Reference Locations

| File | Lines | Purpose |
|------|-------|---------|
| [[`modules/sfp_virustotal.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_virustotal.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L184-L186) | 184–186 | Missing API key validation |
| [[`modules/sfp_virustotal.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_virustotal.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154) | 151–154 | Rate limit (HTTP 204) handling |
| [[`modules/sfp_xforce.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_xforce.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_xforce.py#L215-L216) | 215–216 | Multi-field credential check |
| [[`modules/sfp_zonefiles.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_zonefiles.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_zonefiles.py#L154-L156) | 154–156 | Empty key guard pattern |
| [[`spiderfoot/plugin.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/plugin.py)](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/plugin.py) | — | Base `SpiderFootPlugin` class with `errorState` definition |

## Summary

- **Check credentials first** — Validate `api_key` and related fields in `handleEvent()` before any network request.
- **Use `errorState` as a circuit breaker** — Set `self.errorState = True` on any authentication or fatal error, then return immediately.
- **Inspect HTTP codes and response bodies** — Handle 401/403 for auth failures, 429/204 for throttling, and parse JSON error payloads.
- **Fail fast with clear messages** — Call `self.error()` with actionable descriptions so users understand configuration problems.
- **Prevent redundant requests** — The `errorState` pattern ensures modules stop querying after the first failure, preserving API quotas and scan performance.

## Frequently Asked Questions

### How does SpiderFoot prevent modules from hammering APIs after an authentication failure?

Each module inherits `errorState` from `SpiderFootPlugin`. When set to `True`, the initial guard clause in `handleEvent()` returns immediately, skipping all further event processing for that module during the scan. This single flag prevents redundant network requests and protects API rate limits.

### What HTTP status codes do SpiderFoot modules typically check for authentication errors?

Common checks include **401** (Unauthorized) and **403** (Forbidden) for invalid credentials, plus **429** (Too Many Requests) and **204** (No Content, used by VirusTotal) for throttling. Modules like [[`sfp_virustotal.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_virustotal.py)](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_virustotal.py#L151-L154) demonstrate these patterns explicitly.

### Why do some modules check for API keys in `handleEvent()` rather than `setup()`?

SpiderFoot calls `setup()` once per module initialization, but users may change options during configuration. Checking in `handleEvent()` ensures the latest `opts` values are validated immediately before use. This runtime check catches configuration errors that would otherwise cause silent failures.

### Can I customize error handling for a third-party API with unique error formats?

Yes. Override `handleEvent()` or create a dedicated query method that parses your API's specific error responses—whether JSON fields, HTTP headers, or status codes. Follow the established pattern: log via `self.error()`, set `self.errorState = True`, and return early to halt further processing.