# How to Troubleshoot SpiderFoot Module Failures and Debug Them: Complete Guide

> Fix SpiderFoot module failures by enabling debug logging and examining logs. Learn to identify and resolve syntax errors, API issues, and timeout problems for efficient debugging.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Enable debug logging with `spiderfoot -d` and check `~/.spiderfoot/logs` to identify whether module failures occur during load time (syntax errors in `modules/sfp_*.py`) or runtime (API/timeout issues), then validate the module's `asdict()` output and category definitions against the whitelist in [`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py).**

SpiderFoot is an open-source reconnaissance framework that dynamically loads hundreds of plugin modules at runtime. When troubleshooting SpiderFoot module failures, understanding the architectural flow from [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) through [`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py) to individual `sfp_*.py` modules allows you to pinpoint exactly where things break.

---

## Where SpiderFoot Loads Modules

The module loading pipeline starts in **[`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py)** at line 141:

```python
sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])

```

The actual import logic lives in **[`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py)** → **`loadModulesAsDict`** (lines 21-73). This function:

1. Scans the `modules/` directory for files matching `sfp_*.py`
2. Imports each with `__import__('modules.' + modName, ...)`
3. Instantiates the class and calls **`asdict()`** to extract metadata

Source: [sf.py#L141](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L141) | [helpers.py#L21-L73](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/helpers.py#L21-L73)

---

## Common Module Loading Failures

### Syntax Errors and Import Exceptions

When you see `log.critical("Failed to load modules …")`, the cause is typically a Python exception inside a module's class definition. Run with `-d` (debug) to expose the full traceback.

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| `SyntaxError: Module … has multiple categories defined` | `cats` list contains more than one entry | Change `cats = ["A", "B"]` to `cats = ["A"]` |
| `SyntaxError: Module … has invalid category` | Category not in `valid_categories` whitelist | Use allowed values only (see helpers.py lines 46-49) |
| `ImportError: No module named …` | Missing dependency from [`requirements.txt`](https://github.com/smicallef/spiderfoot/blob/main/requirements.txt) | Run `pip install -r requirements.txt` |

### Validating Module Structure

Every `sfp_*.py` file must satisfy three rules:

1. **Filename matches class name**: [`sfp_example.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_example.py) → `class sfp_example`
2. **`asdict()` returns required keys**:

```python
{
    'name': 'Module Display Name',
    'descr': 'What this module does',
    'provides': ['EVENT_TYPE'],
    'consumes': ['OTHER_EVENT_TYPE'],
    'cats': ['ValidCategory']
}

```

3. **Single valid category**: The `cats` list must contain exactly one string from the whitelist defined in [`helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/helpers.py)

After fixes, verify loading succeeds:

```bash
spiderfoot -M

```

---

## Debugging Runtime Scan Failures

### Launch Point and Execution Flow

Scan execution begins at **[`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py)** lines 235-261 in `start_scan`, which spawns `sfscan.startSpiderFootScanner` as a separate process.

Source: [sf.py#L235-L261](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L235-L261)

### Typical Runtime Error Patterns

| Error Message | Cause | Resolution |
|-------------|-------|------------|
| `You must specify a target …` | Missing `-s` argument | Add `-s example.com` |
| `Could not determine target type …` | Target format unrecognized | Check against `targetTypeFromString` regexes (helpers.py lines 22-38) |
| `Modules enabled …` (empty list) | `-x` strict mode filtered everything | Remove `-x` or adjust `-t` type filters |
| `Failed to initialize database …` | Permission denied on `~/.spiderfoot/spiderfoot.db` | `chmod 700 ~/.spiderfoot` and verify ownership |
| `requests.exceptions.ConnectTimeout` | External API unreachable or no API key | Test endpoint manually; check `~/.spiderfoot/config` for keys |

### Capturing Debug Output

```bash

# Maximum verbosity with debug flag

spiderfoot -d -s example.com -o json > output.json 2> debug.log

# Isolate error lines

grep -i "error\|critical\|exception" debug.log

```

Each scan writes to `~/.spiderfoot/logs/spiderfoot-<timestamp>.log`. Set `__logging = True` in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) line 57 for persistent capture.

---

## Diagnosing Correlation Rule Errors

Correlation rules load at **[`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py)** lines 53-57 via `loadCorrelationRulesRaw`. Failure appears as:

```

log.critical("Failed to load correlation rules: …")

```

Common causes:

- **YAML syntax errors** in `correlations/*.yaml` files
- **Missing required fields** (`description`, `events`)

Validate rule files with an online YAML linter and match structure against repository examples.

---

## Practical Debugging Workflow

```bash

# 1. Run with full debug output

spiderfoot -d -s example.com -o json 2> spiderfoot_debug.log

# 2. Find first failure point

grep -n "Failed to load\|ERROR\|CRITICAL" spiderfoot_debug.log | head -5

# 3. Inspect offending module

sed -n '1,100p' modules/sfp_problematic.py

# 4. Test asdict() output manually

python3 - <<'EOF'
import sys
sys.path.insert(0, '.')
import importlib
mod = importlib.import_module('modules.sfp_problematic')
cls = getattr(mod, 'sfp_problematic')
print(cls().asdict())
EOF

```

For external API issues, set `SPIDERFOOT_LOGS` environment variable to a writable path—modules log request URLs when `_debug` is enabled.

---

## Key Files for Troubleshooting

| File | Purpose |
|------|---------|
| [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) | CLI entry point, module loading orchestration |
| [`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py) | `loadModulesAsDict`, `targetTypeFromString` utilities |
| [`modules/sfp_template.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_template.py) | Reference implementation for module structure |
| [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) | Worker process executing scan modules |
| [`spiderfoot/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/__init__.py) | Core `SpiderFoot` and `SpiderFootDb` classes |
| [`requirements.txt`](https://github.com/smicallef/spiderfoot/blob/main/requirements.txt) | Third-party dependencies |

---

## Summary

- **Use `-d`** for debug stack traces that reveal exact failure points
- **Validate module names** match `sfp_*.py` pattern with corresponding class names
- **Check `cats`** contains exactly one valid category from the whitelist
- **Verify `asdict()`** returns all required metadata keys
- **Confirm target format** matches regex patterns in `targetTypeFromString`
- **Inspect logs** at `~/.spiderfoot/logs/` for module-level details
- **Test dependencies** from [`requirements.txt`](https://github.com/smicallef/spiderfoot/blob/main/requirements.txt) when import errors occur

---

## Frequently Asked Questions

### Why does SpiderFoot fail to load any modules?

The most common cause is a syntax error in one `sfp_*.py` file that aborts the entire `loadModulesAsDict` operation. Run `spiderfoot -d -M` to see which module triggers the exception, then inspect that file for malformed `cats` definitions or `asdict()` implementation errors.

### How do I find which module is causing a scan to hang?

Enable debug logging with `-d` and check `~/.spiderfoot/logs/` for the last module activity before the hang. Network timeouts in modules using `requests` without proper timeout handling are typical culprits—look for `sfp_*.py` files making HTTP calls and add explicit timeout parameters.

### What causes "invalid category" errors in custom modules?

SpiderFoot enforces a whitelist of categories in [`helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/helpers.py) lines 46-49. Custom modules must use a single allowed string in their `cats` list. Multiple categories or misspelled category names trigger `SyntaxError` during the `loadModulesAsDict` import phase.

### Where are API keys and configuration stored for troubleshooting?

SpiderFoot stores configuration in `~/.spiderfoot/config` as a SQLite database. Module-specific API keys and settings persist here. If a module fails due to authentication, verify the key exists in this file or re-enter it through the web UI under **Settings**.