How to Debug Module Failures and Handle Errors in SpiderFoot

Enable verbose logging with -d, check ~/.spiderfoot/logs for stack traces, and validate that modules have correct filenames, single categories, and properly formatted asdict() methods.

SpiderFoot is an open-source reconnaissance framework that dynamically loads hundreds of plugin modules at runtime. When modules fail to load, crash during execution, or produce unexpected results, understanding the underlying architecture is essential for rapid diagnosis. This guide walks through the exact locations in the source code where failures occur and provides specific steps to debug module failures and handle errors in SpiderFoot.

Where Module Failures Originate

SpiderFoot's architecture separates concerns across several key components. Knowing which layer produces an error lets you focus debugging effort efficiently.

Component	Responsibility	Typical Failure Points
sf.py – Main entry point	Parses CLI arguments, creates global config, loads modules, starts scans or web UI	Invalid CLI options, missing config values, web server startup failures
spiderfoot/helpers.py	Provides loadModulesAsDict() which imports each module under modules/	Syntax errors, duplicate categories, missing sfp_ prefix, malformed asdict()
Individual modules (modules/sfp_*.py)	Contain data-gathering logic; must expose asdict() and implement required interface	Runtime exceptions, API key issues, incorrect provides/consumes definitions
SpiderFootDb	Persists scan results and configuration	File permission problems, corrupted schema
Logging subsystem	Collects debug/error information	Disabled logging, missing log directory

Debugging Module Loading Failures

How Modules Are Loaded

In sf.py at line 141, modules are imported via:

sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])

The actual import logic resides in spiderfoot/helpers.py → loadModulesAsDict (lines 21-73). This function:

1. Scans modules/ for files matching sfp_*.py
2. Imports each with __import__('modules.' + modName, ...)
3. Instantiates the class and calls asdict() to validate structure

Source: sf.py#L141 and helpers.py#L21-L73

Common Loading Errors and Fixes

Symptom	Root Cause	Solution
log.critical("Failed to load modules …")	Syntax error or exception in module class definition	Run with -d flag and examine traceback in console or ~/.spiderfoot/logs
SyntaxError: Module … has multiple categories defined	cats list contains more than one entry	Change cats = ["A", "B"] to cats = ["Category"] — single string only
SyntaxError: Module … has invalid category	Category not in valid_categories whitelist	Match one of the allowed values (helpers.py lines 46-49)
ImportError: No module named …	Missing third-party dependency	Install from requirements.txt: pip install -r requirements.txt

To verify a module loads correctly after fixes:

spiderfoot -M

This lists all successfully loaded modules.

Validating Module Structure

Each module must satisfy three requirements:

1. Filename matches class name: sfp_example.py → class sfp_example
2. asdict() returns required keys: name, descr, provides, consumes, cats
3. Single valid category: Must appear in valid_categories (e.g., "DNS", "SOCIAL")

Quick validation command:

python -c "
import importlib
mod = importlib.import_module('modules.sfp_example')
obj = getattr(mod, 'sfp_example')()
print(obj.asdict())
"

If this raises an exception, the asdict() method needs correction.

Debugging Runtime Scan Failures

Where Scans Launch

The start_scan function in sf.py (lines 235-261) creates a SpiderFoot instance and spawns sfscan.startSpiderFootScanner in a separate process.

Source: sf.py#L235-L261

Common Runtime Issues

Error Message	Cause	Fix
You must specify a target	Missing -s argument	Add target: spiderfoot -s example.com
Could not determine target type	Target format unrecognized	Use valid format per targetTypeFromString (helpers.py lines 22-38): IP, domain, CIDR, ASN, email, phone, username, or bitcoin address
Modules enabled … shows empty list	-x (strict mode) filtered all modules	Remove -x or adjust -t to include compatible event types
Failed to initialize database	Permission denied on ~/.spiderfoot/spiderfoot.db	Create directory with mkdir -p ~/.spiderfoot && chmod 700 ~/.spiderfoot
requests.exceptions.ConnectTimeout	External API unreachable or missing API key	Test endpoint manually; store keys in ~/.spiderfoot/config

Essential Debug Flags

• -d: Enable debug output (sets _debug = True internally)
• -o json: Structured output for log parsing
• -max-threads 1: Reduce concurrency to simplify traceback analysis

Debug execution example:

spiderfoot -d -s example.com -o json 2> spiderfoot_debug.log

Log files are stored in ~/.spiderfoot/logs/ with timestamps: spiderfoot-<timestamp>.log

Debugging Correlation Rule Errors

Correlation rules load in sf.py (lines 53-57) via loadCorrelationRulesRaw. Failures appear as:

log.critical("Failed to load correlation rules: …")

Common causes:

• YAML syntax errors in correlations/*.yaml files
• Missing required fields: description, events

Validate rule files with any YAML linter and compare against repository examples.

Step-by-Step Debugging Workflow

# 1. Run with maximum verbosity, capture all output

spiderfoot -d -s example.com -o json > scan_output.json 2> spiderfoot_debug.log

# 2. Find first critical error in logs

grep -i "error\|critical\|failed" spiderfoot_debug.log | head -5

# 3. Inspect offending module (example: sfp_dnsresolve)

sed -n '1,120p' modules/sfp_dnsresolve.py

# 4. Validate asdict() returns proper structure

python - <<'PY'
import sys
sys.path.insert(0, '.')
import importlib
mod = importlib.import_module('modules.sfp_dnsresolve')
cls = getattr(mod, 'sfp_dnsresolve')
print(cls().asdict())
PY

# 5. Check database permissions if DB errors occur

ls -la ~/.spiderfoot/spiderfoot.db
sqlite3 ~/.spiderfoot/spiderfoot.db ".tables"

For HTTP-related failures, set SPIDERFOOT_LOGS environment variable to a writable directory. Most modules log request URLs when _debug is enabled.

Key Source Files for Debugging

File	Purpose	Direct Link
sf.py	CLI driver, module loading, scan orchestration	sf.py
spiderfoot/helpers.py	loadModulesAsDict(), targetTypeFromString()	helpers.py
modules/sfp_template.py	Reference module skeleton	sfp_template.py
sfscan.py	Worker process running selected modules	sfscan.py
spiderfoot/__init__.py	Core SpiderFoot and SpiderFootDb classes	spiderfoot/init.py
requirements.txt	Third-party dependencies	requirements.txt

Summary

• Run SpiderFoot with -d to capture full stack traces in ~/.spiderfoot/logs
• Ensure module filenames start with sfp_ and class names match exactly
• Verify cats contains one valid category from the whitelist
• Confirm asdict() returns a dictionary with name, descr, provides, consumes, and cats
• Check target strings against targetTypeFromString regex patterns
• Validate database file permissions and directory ownership
• Use -max-threads 1 to reduce noise in concurrent failure scenarios

Frequently Asked Questions

Where does SpiderFoot store debug logs?

SpiderFoot writes logs to ~/.spiderfoot/logs/ with filenames following the pattern spiderfoot-<timestamp>.log. When running with -d, debug output also appears on stderr. Set the SPIDERFOOT_LOGS environment variable to override the log directory location.

Why does my custom module fail to load with "multiple categories defined"?

The loadModulesAsDict() function in spiderfoot/helpers.py enforces a single category per module. Change your module's cats variable from a list with multiple strings (["DNS", "SEARCH"]) to a single-element list (["DNS"]). The whitelist of valid categories is defined at helpers.py lines 46-49.

How do I identify which module is causing a scan to crash?

Run with -d -max-threads 1 to serialize execution and capture un-interleaved tracebacks. Examine the log for the last module name printed before the exception, or search for Module [sfp_ to find module start/end markers. Most modules log entry points when _debug is enabled.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}