How to Troubleshoot SpiderFoot Module Failures and Debug Them: Complete Guide

Enable debug logging with spiderfoot -d and check ~/.spiderfoot/logs to identify whether module failures occur during load time (syntax errors in modules/sfp_*.py) or runtime (API/timeout issues), then validate the module's asdict() output and category definitions against the whitelist in spiderfoot/helpers.py.

SpiderFoot is an open-source reconnaissance framework that dynamically loads hundreds of plugin modules at runtime. When troubleshooting SpiderFoot module failures, understanding the architectural flow from sf.py through spiderfoot/helpers.py to individual sfp_*.py modules allows you to pinpoint exactly where things break.

---

Where SpiderFoot Loads Modules

The module loading pipeline starts in sf.py at line 141:

sfModules = SpiderFootHelpers.loadModulesAsDict(mod_dir, ['sfp_template.py'])

The actual import logic lives in spiderfoot/helpers.py → loadModulesAsDict (lines 21-73). This function:

1. Scans the modules/ directory for files matching sfp_*.py
2. Imports each with __import__('modules.' + modName, ...)
3. Instantiates the class and calls asdict() to extract metadata

Source: sf.py#L141 | helpers.py#L21-L73

---

Common Module Loading Failures

Syntax Errors and Import Exceptions

When you see log.critical("Failed to load modules …"), the cause is typically a Python exception inside a module's class definition. Run with -d (debug) to expose the full traceback.

Symptom	Root Cause	Fix
SyntaxError: Module … has multiple categories defined	cats list contains more than one entry	Change cats = ["A", "B"] to cats = ["A"]
SyntaxError: Module … has invalid category	Category not in valid_categories whitelist	Use allowed values only (see helpers.py lines 46-49)
ImportError: No module named …	Missing dependency from requirements.txt	Run pip install -r requirements.txt

Validating Module Structure

Every sfp_*.py file must satisfy three rules:

1. Filename matches class name: sfp_example.py → class sfp_example
2. asdict() returns required keys:

{
    'name': 'Module Display Name',
    'descr': 'What this module does',
    'provides': ['EVENT_TYPE'],
    'consumes': ['OTHER_EVENT_TYPE'],
    'cats': ['ValidCategory']
}

3. Single valid category: The cats list must contain exactly one string from the whitelist defined in helpers.py

After fixes, verify loading succeeds:

spiderfoot -M

---

Debugging Runtime Scan Failures

Launch Point and Execution Flow

Scan execution begins at sf.py lines 235-261 in start_scan, which spawns sfscan.startSpiderFootScanner as a separate process.

Source: sf.py#L235-L261

Typical Runtime Error Patterns

Error Message	Cause	Resolution
You must specify a target …	Missing -s argument	Add -s example.com
Could not determine target type …	Target format unrecognized	Check against targetTypeFromString regexes (helpers.py lines 22-38)
Modules enabled … (empty list)	-x strict mode filtered everything	Remove -x or adjust -t type filters
Failed to initialize database …	Permission denied on ~/.spiderfoot/spiderfoot.db	chmod 700 ~/.spiderfoot and verify ownership
requests.exceptions.ConnectTimeout	External API unreachable or no API key	Test endpoint manually; check ~/.spiderfoot/config for keys

Capturing Debug Output

# Maximum verbosity with debug flag

spiderfoot -d -s example.com -o json > output.json 2> debug.log

# Isolate error lines

grep -i "error\|critical\|exception" debug.log

Each scan writes to ~/.spiderfoot/logs/spiderfoot-<timestamp>.log. Set __logging = True in sf.py line 57 for persistent capture.

---

Diagnosing Correlation Rule Errors

Correlation rules load at sf.py lines 53-57 via loadCorrelationRulesRaw. Failure appears as:

log.critical("Failed to load correlation rules: …")

Common causes:

• YAML syntax errors in correlations/*.yaml files
• Missing required fields (description, events)

Validate rule files with an online YAML linter and match structure against repository examples.

---

Practical Debugging Workflow

# 1. Run with full debug output

spiderfoot -d -s example.com -o json 2> spiderfoot_debug.log

# 2. Find first failure point

grep -n "Failed to load\|ERROR\|CRITICAL" spiderfoot_debug.log | head -5

# 3. Inspect offending module

sed -n '1,100p' modules/sfp_problematic.py

# 4. Test asdict() output manually

python3 - <<'EOF'
import sys
sys.path.insert(0, '.')
import importlib
mod = importlib.import_module('modules.sfp_problematic')
cls = getattr(mod, 'sfp_problematic')
print(cls().asdict())
EOF

For external API issues, set SPIDERFOOT_LOGS environment variable to a writable path—modules log request URLs when _debug is enabled.

---

Key Files for Troubleshooting

File	Purpose
sf.py	CLI entry point, module loading orchestration
spiderfoot/helpers.py	loadModulesAsDict, targetTypeFromString utilities
modules/sfp_template.py	Reference implementation for module structure
sfscan.py	Worker process executing scan modules
spiderfoot/__init__.py	Core SpiderFoot and SpiderFootDb classes
requirements.txt	Third-party dependencies

---

Summary

• Use -d for debug stack traces that reveal exact failure points
• Validate module names match sfp_*.py pattern with corresponding class names
• Check cats contains exactly one valid category from the whitelist
• Verify asdict() returns all required metadata keys
• Confirm target format matches regex patterns in targetTypeFromString
• Inspect logs at ~/.spiderfoot/logs/ for module-level details
• Test dependencies from requirements.txt when import errors occur

---

Frequently Asked Questions

Why does SpiderFoot fail to load any modules?

The most common cause is a syntax error in one sfp_*.py file that aborts the entire loadModulesAsDict operation. Run spiderfoot -d -M to see which module triggers the exception, then inspect that file for malformed cats definitions or asdict() implementation errors.

How do I find which module is causing a scan to hang?

Enable debug logging with -d and check ~/.spiderfoot/logs/ for the last module activity before the hang. Network timeouts in modules using requests without proper timeout handling are typical culprits—look for sfp_*.py files making HTTP calls and add explicit timeout parameters.

What causes "invalid category" errors in custom modules?

SpiderFoot enforces a whitelist of categories in helpers.py lines 46-49. Custom modules must use a single allowed string in their cats list. Multiple categories or misspelled category names trigger SyntaxError during the loadModulesAsDict import phase.

Where are API keys and configuration stored for troubleshooting?

SpiderFoot stores configuration in ~/.spiderfoot/config as a SQLite database. Module-specific API keys and settings persist here. If a module fails due to authentication, verify the key exists in this file or re-enter it through the web UI under Settings.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}