# How to Debug Module Failures and Handle Errors in SpiderFoot

> Debug SpiderFoot module failures and handle errors effectively. Learn to enable verbose logging, check logs for stack traces, and validate module configurations for seamless operation.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Enable verbose logging with `-d`, check `~/.spiderfoot/logs` for stack traces, and validate that modules have correct filenames, single categories, and properly formatted `asdict()` methods.**

SpiderFoot is an open-source reconnaissance framework that dynamically loads hundreds of plugin modules at runtime. When modules fail to load, crash during execution, or produce unexpected results, understanding the underlying architecture is essential for rapid diagnosis. This guide walks through the exact locations in the source code where failures occur and provides specific steps to debug module failures and handle errors in SpiderFoot.

## Where Module Failures Originate

SpiderFoot's architecture separates concerns across several key components. Knowing which layer produces an error lets you focus debugging effort efficiently.

| Component | Responsibility | Typical Failure Points |
|-----------|---------------|------------------------|
| **[`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py)** – Main entry point | Parses CLI arguments, creates global config, loads modules, starts scans or web UI | Invalid CLI options, missing config values, web server startup failures |
| **[`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py)** | Provides `loadModulesAsDict()` which imports each module under `modules/` | Syntax errors, duplicate categories, missing `sfp_` prefix, malformed `asdict()` |
| **Individual modules (`modules/sfp_*.py`)** | Contain data-gathering logic; must expose `asdict()` and implement required interface | Runtime exceptions, API key issues, incorrect `provides`/`consumes` definitions |
| **`SpiderFootDb`** | Persists scan results and configuration | File permission problems, corrupted schema |
| **Logging subsystem** | Collects debug/error information | Disabled logging, missing log directory |

## Debugging Module Loading Failures

### How Modules Are Loaded

In [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) at line 141, modules are imported via:

```python
sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])

```

The actual import logic resides in **[`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py)** → **`loadModulesAsDict`** (lines 21-73). This function:

1. Scans `modules/` for files matching `sfp_*.py`
2. Imports each with `__import__('modules.' + modName, ...)`
3. Instantiates the class and calls `asdict()` to validate structure

Source: [sf.py#L141](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L141) and [helpers.py#L21-L73](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/helpers.py#L21-L73)

### Common Loading Errors and Fixes

| Symptom | Root Cause | Solution |
|---------|-----------|----------|
| `log.critical("Failed to load modules …")` | Syntax error or exception in module class definition | Run with `-d` flag and examine traceback in console or `~/.spiderfoot/logs` |
| `SyntaxError: Module … has multiple categories defined` | `cats` list contains more than one entry | Change `cats = ["A", "B"]` to `cats = ["Category"]` — single string only |
| `SyntaxError: Module … has invalid category` | Category not in `valid_categories` whitelist | Match one of the allowed values (helpers.py lines 46-49) |
| `ImportError: No module named …` | Missing third-party dependency | Install from [`requirements.txt`](https://github.com/smicallef/spiderfoot/blob/main/requirements.txt): `pip install -r requirements.txt` |

To verify a module loads correctly after fixes:

```bash
spiderfoot -M

```

This lists all successfully loaded modules.

### Validating Module Structure

Each module must satisfy three requirements:

1. **Filename matches class name**: [`sfp_example.py`](https://github.com/smicallef/spiderfoot/blob/main/sfp_example.py) → `class sfp_example`
2. **`asdict()` returns required keys**: `name`, `descr`, `provides`, `consumes`, `cats`
3. **Single valid category**: Must appear in `valid_categories` (e.g., `"DNS"`, `"SOCIAL"`)

Quick validation command:

```python
python -c "
import importlib
mod = importlib.import_module('modules.sfp_example')
obj = getattr(mod, 'sfp_example')()
print(obj.asdict())
"

```

If this raises an exception, the `asdict()` method needs correction.

## Debugging Runtime Scan Failures

### Where Scans Launch

The `start_scan` function in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) (lines 235-261) creates a `SpiderFoot` instance and spawns `sfscan.startSpiderFootScanner` in a separate process.

Source: [sf.py#L235-L261](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L235-L261)

### Common Runtime Issues

| Error Message | Cause | Fix |
|-------------|-------|-----|
| `You must specify a target` | Missing `-s` argument | Add target: `spiderfoot -s example.com` |
| `Could not determine target type` | Target format unrecognized | Use valid format per `targetTypeFromString` (helpers.py lines 22-38): IP, domain, CIDR, ASN, email, phone, username, or bitcoin address |
| `Modules enabled …` shows empty list | `-x` (strict mode) filtered all modules | Remove `-x` or adjust `-t` to include compatible event types |
| `Failed to initialize database` | Permission denied on `~/.spiderfoot/spiderfoot.db` | Create directory with `mkdir -p ~/.spiderfoot && chmod 700 ~/.spiderfoot` |
| `requests.exceptions.ConnectTimeout` | External API unreachable or missing API key | Test endpoint manually; store keys in `~/.spiderfoot/config` |

### Essential Debug Flags

- **`-d`**: Enable debug output (sets `_debug = True` internally)
- **`-o json`**: Structured output for log parsing
- **`-max-threads 1`**: Reduce concurrency to simplify traceback analysis

Debug execution example:

```bash
spiderfoot -d -s example.com -o json 2> spiderfoot_debug.log

```

Log files are stored in `~/.spiderfoot/logs/` with timestamps: `spiderfoot-<timestamp>.log`

## Debugging Correlation Rule Errors

Correlation rules load in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) (lines 53-57) via `loadCorrelationRulesRaw`. Failures appear as:

```

log.critical("Failed to load correlation rules: …")

```

Common causes:

- YAML syntax errors in `correlations/*.yaml` files
- Missing required fields: `description`, `events`

Validate rule files with any YAML linter and compare against repository examples.

## Step-by-Step Debugging Workflow

```bash

# 1. Run with maximum verbosity, capture all output

spiderfoot -d -s example.com -o json > scan_output.json 2> spiderfoot_debug.log

# 2. Find first critical error in logs

grep -i "error\|critical\|failed" spiderfoot_debug.log | head -5

# 3. Inspect offending module (example: sfp_dnsresolve)

sed -n '1,120p' modules/sfp_dnsresolve.py

# 4. Validate asdict() returns proper structure

python - <<'PY'
import sys
sys.path.insert(0, '.')
import importlib
mod = importlib.import_module('modules.sfp_dnsresolve')
cls = getattr(mod, 'sfp_dnsresolve')
print(cls().asdict())
PY

# 5. Check database permissions if DB errors occur

ls -la ~/.spiderfoot/spiderfoot.db
sqlite3 ~/.spiderfoot/spiderfoot.db ".tables"

```

For HTTP-related failures, set `SPIDERFOOT_LOGS` environment variable to a writable directory. Most modules log request URLs when `_debug` is enabled.

## Key Source Files for Debugging

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) | CLI driver, module loading, scan orchestration | [sf.py](https://github.com/smicallef/spiderfoot/blob/master/sf.py) |
| [`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py) | `loadModulesAsDict()`, `targetTypeFromString()` | [helpers.py](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/helpers.py) |
| [`modules/sfp_template.py`](https://github.com/smicallef/spiderfoot/blob/main/modules/sfp_template.py) | Reference module skeleton | [sfp_template.py](https://github.com/smicallef/spiderfoot/blob/master/modules/sfp_template.py) |
| [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) | Worker process running selected modules | [sfscan.py](https://github.com/smicallef/spiderfoot/blob/master/sfscan.py) |
| [`spiderfoot/__init__.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/__init__.py) | Core `SpiderFoot` and `SpiderFootDb` classes | [spiderfoot/__init__.py](https://github.com/smicallef/spiderfoot/blob/master/spiderfoot/__init__.py) |
| [`requirements.txt`](https://github.com/smicallef/spiderfoot/blob/main/requirements.txt) | Third-party dependencies | [requirements.txt](https://github.com/smicallef/spiderfoot/blob/master/requirements.txt) |

## Summary

- Run SpiderFoot with `-d` to capture full stack traces in `~/.spiderfoot/logs`
- Ensure module filenames start with `sfp_` and class names match exactly
- Verify `cats` contains one valid category from the whitelist
- Confirm `asdict()` returns a dictionary with `name`, `descr`, `provides`, `consumes`, and `cats`
- Check target strings against `targetTypeFromString` regex patterns
- Validate database file permissions and directory ownership
- Use `-max-threads 1` to reduce noise in concurrent failure scenarios

## Frequently Asked Questions

### Where does SpiderFoot store debug logs?

SpiderFoot writes logs to `~/.spiderfoot/logs/` with filenames following the pattern `spiderfoot-<timestamp>.log`. When running with `-d`, debug output also appears on stderr. Set the `SPIDERFOOT_LOGS` environment variable to override the log directory location.

### Why does my custom module fail to load with "multiple categories defined"?

The `loadModulesAsDict()` function in [`spiderfoot/helpers.py`](https://github.com/smicallef/spiderfoot/blob/main/spiderfoot/helpers.py) enforces a single category per module. Change your module's `cats` variable from a list with multiple strings (`["DNS", "SEARCH"]`) to a single-element list (`["DNS"]`). The whitelist of valid categories is defined at helpers.py lines 46-49.

### How do I identify which module is causing a scan to crash?

Run with `-d -max-threads 1` to serialize execution and capture un-interleaved tracebacks. Examine the log for the last module name printed before the exception, or search for `Module [sfp_` to find module start/end markers. Most modules log entry points when `_debug` is enabled.