How to Add Custom Threat Intelligence Feeds to SpiderFoot: A Complete Guide

Use the built-in sfp_customfeed module to import any JSON-based indicator list into SpiderFoot without modifying core code.

SpiderFoot ships with a generic Custom Feed plugin that transforms your own threat intelligence into first-class scan events. Located at modules/sfp_customfeed.py in the smicallef/spiderfoot repository, this plugin reads a JSON file of indicators and injects them into SpiderFoot's event pipeline for correlation and enrichment alongside data from 200+ built-in modules.

How the Custom Feed Plugin Works

The sfp_customfeed module implements the standard SpiderFootPlugin interface, giving your custom indicators the same capabilities as commercial threat feeds. Here's the execution flow based on the source code:

1. Configuration loading — Reads the customfeed option from scan profile or CLI (lines 19-28)
2. JSON parsing — Validates and loads the feed file using json.load(); malformed JSON aborts with a clear error (lines 31-38)
3. Event creation — Calls self.sf.createEvent() for each indicator, mapping type and value plus optional metadata (lines 59-77)
4. Event publication — Adds events to the global queue for consumption by correlation modules (lines 78-82)

Because the plugin uses SpiderFoot's internal self.sf API, your custom indicators automatically participate in cross-module correlation, enrichment chaining, and report generation without any code changes.

Required JSON Format for Custom Feeds

Each indicator object requires two mandatory fields and supports three optional metadata fields:

Field	Required	Description
type	✅	SpiderFoot event type: ipv4, ipv6, domain, hostname, email, url, hash, etc.
value	✅	Raw indicator string (IP address, domain name, hash, etc.)
source	❌	Custom provenance label for tracking (appears in event metadata)
confidence	❌	Integer 0-100 representing confidence score
description	❌	Human-readable context about the indicator

Example Custom Feed File

Create a file named myfeed.json:

[
  {
    "type": "ipv4",
    "value": "198.51.100.23",
    "source": "MyInternalIPList",
    "confidence": 95,
    "description": "Known compromised host from SOC alert"
  },
  {
    "type": "domain",
    "value": "malicious.example.com",
    "source": "InternalPhishingFeed",
    "confidence": 80,
    "description": "Phishing landing page identified by abuse desk"
  },
  {
    "type": "email",
    "value": "badguy@evil.org",
    "source": "CompromisedCredentials",
    "confidence": 90,
    "description": "Credential from 2023 breach corpus"
  },
  {
    "type": "hash",
    "value": "d41d8cd98f00b204e9800998ecf8427e",
    "source": "MalwareLab",
    "confidence": 100,
    "description": "Confirmed Lazarus group sample"
  }
]

Enabling the Custom Feed Module

Method 1: Scan Profile Configuration

Add to your profile JSON (e.g., profiles/custom-intel.json):

{
  "modules": {
    "sfp_customfeed": {
      "enabled": true,
      "customfeed": "/path/to/myfeed.json"
    },
    "sfp_dnsresolve": {
      "enabled": true
    },
    "sfp_whois": {
      "enabled": true
    }
  }
}

Run with:

python sf.py -s corp-target.com -p profiles/custom-intel.json

Method 2: Command-Line Options

python sf.py -s corp-target.com \
    -m sfp_customfeed,sfp_dnsresolve \
    -o customfeed=/path/to/myfeed.json

Method 3: Docker Deployment

docker run --rm \
    -v $(pwd)/myfeed.json:/data/myfeed.json:ro \
    spiderfoot/spiderfoot \
    -s corp-target.com \
    -m sfp_customfeed \
    -o customfeed=/data/myfeed.json

Verifying Custom Feed Integration

During scan startup, monitor logs for confirmation:

[+] SpiderFoot 3.5.0 initialized
[+] Loaded 201 modules
[+] sfp_customfeed - Loaded 4 indicators from /path/to/myfeed.json
[+] sfp_customfeed - Created event ipv4:198.51.100.23 (source: MyInternalIPList, confidence: 95)
[+] sfp_customfeed - Created event domain:malicious.example.com (source: InternalPhishingFeed, confidence: 80)
[+] sfp_customfeed - Created event email:badguy@evil.org (source: CompromisedCredentials, confidence: 90)
[+] sfp_customfeed - Created event hash:d41d8cd98f00b204e9800998ecf8427e (source: MalwareLab, confidence: 100)

Your custom indicators now flow through SpiderFoot's standard pipeline. The sfp_dnsresolve module will resolve IP-domain relationships, sfp_abusech will check blocklists, and sfp_correlations will flag matches against other discovered assets.

Programmatic Access to Custom Events

When embedding SpiderFoot as a library, retrieve custom events by source label:

import spiderfoot.sf as sf

engine = sf.SpiderFootEngine()
engine.setTarget('corp-target.com')
engine.enableModule('sfp_customfeed')
engine.setOption('sfp_customfeed.customfeed', '/path/to/myfeed.json')
engine.start()

# Extract all events from scan

all_events = engine.getEvents()

# Filter for your custom feed sources

intel_sources = {'MyInternalIPList', 'InternalPhishingFeed', 'CompromisedCredentials', 'MalwareLab'}
custom_intel = [
    e for e in all_events 
    if e.get('source') in intel_sources
]

# Group by indicator type

from collections import defaultdict
by_type = defaultdict(list)
for event in custom_intel:
    by_type[event.get('type')].append(event.get('value'))

print(f"Loaded {len(custom_intel)} custom indicators:")
for evt_type, values in by_type.items():
    print(f"  {evt_type}: {len(values)} indicators")

Key Files and Source References

File	Purpose	Lines
modules/sfp_customfeed.py	Core plugin implementation	19-82
spiderfoot/event.py	Event class instantiated by plugin	Full file
spiderfoot/plugin.py	Base SpiderFootPlugin class	Full file
sf.py	CLI entry point for scan orchestration	Full file

Supported Event Types for Custom Feeds

The type field accepts any valid SpiderFoot event type. Common values include:

• Network: ipv4, ipv6, domain, hostname, url, netblock
• Identity: email, username, person_name, phone_number
• Threat: hash, hash_md5, hash_sha1, hash_sha256, malware_name
• Infrastructure: port, banner, geoip, certificate
• Content: raw_rfi, raw_sql_injection, raw_xss

Consult spiderfoot/event.py for the complete enumeration.

Summary

• Prepare JSON — Create a structured indicator file with type and value required fields
• Enable plugin — Activate sfp_customfeed via scan profile or CLI option customfeed=
• Run scan — Indicators automatically become SpiderFoot events for correlation and reporting
• No core modifications — The plugin uses standard SpiderFootPlugin interface and self.sf.createEvent() API

Your custom threat intelligence receives identical treatment to commercial feeds, enabling seamless integration with SpiderFoot's enrichment, correlation, and visualization capabilities.

Frequently Asked Questions

What happens if my JSON file is malformed?

SpiderFoot aborts the scan immediately with a descriptive error. The plugin wraps json.load() in exception handling at lines 31-38 of sfp_customfeed.py, surfacing parse errors like JSONDecodeError: Expecting property name enclosed in double quotes before any events are generated.

Can I use multiple custom feed files in one scan?

Yes. Enable multiple instances by creating separate scan profiles or running sequential scans with different -o customfeed= values. Each scan loads its specified feed independently; there is no built-in merging of multiple JSON files within a single plugin instance.

Do custom feed indicators trigger automatic enrichment?

Absolutely. Once sfp_customfeed publishes events to the global queue, any enabled module observing those event types will process them. A domain indicator will trigger sfp_dnsresolve, sfp_whoxy, and sfp_abusech if those modules are active in your scan configuration.

Is there a size limit for custom feed files?

No hardcoded limit exists in the plugin source. However, extremely large files (100K+ indicators) may impact startup time and memory usage since the entire JSON array loads into memory before event generation begins. For production-scale feeds, consider splitting into multiple scans or implementing a custom streaming plugin using the same SpiderFootPlugin base class.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}