How to Customize User Agent Strings for HTTP Requests in SpiderFoot

SpiderFoot allows you to override the default User-Agent header globally using the _useragent configuration option, which supports static strings, local file references prefixed with @, or remote URLs containing lists of agents.

SpiderFoot sends all HTTP requests with a User-Agent header that identifies the client to target servers. The open-source OSINT tool provides flexible customization options for this header through its global configuration system. This guide explains how to customize user agent strings for HTTP requests in SpiderFoot based on the actual source code implementation in the smicallef/spiderfoot repository.

Default User Agent Configuration

SpiderFoot defines its default User-Agent in sf.py within the global configuration dictionary:

'_useragent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0'

This default masquerades as Firefox 62 on Windows 10 (see lines 59-60 in [sf.py](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L59-L60)).

The same file documents three supported input formats for customizing this value (lines 78-79):

1. Direct string — Provide any custom User-Agent text
2. Local file with @ prefix — SpiderFoot randomly selects from a file containing one agent per line
3. Remote URL — SpiderFoot downloads a plain-text list and randomly selects from it

How the Custom Value Propagates Through the Codebase

Configuration Processing in sfscan.py

When a scan initializes, sfscan.py transforms the raw configuration value into its final usable form. Line 196 invokes optValueToData to expand file references or fetch remote lists:

self.__config['_useragent'] = self.__sf.optValueToData(self.__config['_useragent'])

This method handles the @file syntax and HTTP(S) URL resolution before any modules execute.

Header Application in sflib.py

Every HTTP request flows through SpiderFoot.fetchUrl in sflib.py. Lines 558-562 implement the selection logic:

if isinstance(useragent, list):
    header['User-Agent'] = random.SystemRandom().choice(useragent)
else:
    header['User-Agent'] = useragent

This ensures that whether you provide a single string or a list, the correct header format reaches the target server.

Module-Level Access

All SpiderFoot modules receive the processed value via self.opts['_useragent'] or self.sf.opts['_useragent']. Modules pass this directly to fetchUrl, meaning your customization affects every HTTP request across the entire scan without per-module configuration.

Practical Customization Examples

Set a Static Custom User Agent

Create a configuration file with your specific agent string:

cat > custom_ua.json <<EOF
{
    "_useragent": "MySpiderFoot/1.0 (+https://example.com/bot)"
}
EOF

Execute the scan with this configuration:

python3 sf.py -s example.com -c custom_ua.json

Use Random Agents from a Local File

Prepare a file with one User-Agent per line:

cat > useragents.txt <<EOF
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.0 Chrome/108.0.0.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Safari/605.1.15
Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0
EOF

Reference it with the @ prefix:

cat > random_local.json <<EOF
{
    "_useragent": "@$(pwd)/useragents.txt"
}
EOF

python3 sf.py -s example.com -c random_local.json

Fetch Random Agents from a Remote URL

Point SpiderFoot to an external list:

cat > random_remote.json <<EOF
{
    "_useragent": "https://raw.githubusercontent.com/marcusbotacin/ua-list/master/ua-list.txt"
}
EOF

python3 sf.py -s example.com -c random_remote.json

SpiderFoot downloads the list once at scan start and randomly selects for each request.

Programmatic Override for Individual Modules

For specialized testing, override the User-Agent for a specific module instance:

from spiderfoot import SpiderFoot

sf = SpiderFoot()

# Customize only the DNSDumpster module

sf.modules['sfp_dnsdumpster'].opts['_useragent'] = "CustomAgent/2.0 SecurityResearch"

sf.modules['sfp_dnsdumpster'].start()

This technique bypasses the global configuration for targeted testing scenarios.

Key Files in the User Agent Implementation

File	Location	Purpose
sf.py	Lines 59-78	Defines default value and documents the three input formats
sfscan.py	Line 196	Processes @file and URL references at scan initialization
sflib.py	Lines 558-562	Applies final User-Agent header in fetchUrl
Module files	Various	Inherit via self.opts['_useragent'] for all requests

Summary

• SpiderFoot's _useragent configuration option controls the User-Agent header for all HTTP requests
• Three input formats supported: plain string, @file reference, or remote URL
• Random selection occurs via random.SystemRandom().choice when a list is provided
• Global propagation ensures consistency across every module without individual configuration
• Source locations: sf.py for defaults, sfscan.py for preprocessing, sflib.py for application

Frequently Asked Questions

What is SpiderFoot's default User-Agent string?

SpiderFoot uses Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0 as defined in sf.py lines 59-60. This Firefox 62 on Windows 10 signature helps reduce detection by target servers expecting automated tools.

How does SpiderFoot handle multiple User-Agents?

When you provide a file path prefixed with @ or a remote URL to a text list, sfscan.py converts this into a Python list via optValueToData. Then in sflib.py lines 558-562, random.SystemRandom().choice(useragent) selects a random entry for each HTTP request.

Can different modules use different User-Agents simultaneously?

Yes, but only through programmatic manipulation. While the global configuration applies universally, you can override self.opts['_useragent'] on individual module instances before calling start(). There is no built-in per-module configuration in the CLI or JSON config system.

Where does the remote User-Agent list get downloaded?

The optValueToData method in sflib.py handles URL fetching during scan initialization in sfscan.py. The download occurs once when the scan starts, not per-request, and the resulting list remains in memory for random selection throughout the scan lifetime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}