# How to Customize User Agent Strings for HTTP Requests in SpiderFoot

> Learn how to customize User Agent strings for SpiderFoot HTTP requests. Override default agents globally with static strings, local files, or remote URLs.

- Repository: [Steve Micallef/spiderfoot](https://github.com/smicallef/spiderfoot)
- Tags: how-to-guide
- Published: 2026-08-15

---

**SpiderFoot allows you to override the default User-Agent header globally using the `_useragent` configuration option, which supports static strings, local file references prefixed with `@`, or remote URLs containing lists of agents.**

SpiderFoot sends all HTTP requests with a **User-Agent** header that identifies the client to target servers. The open-source OSINT tool provides flexible customization options for this header through its global configuration system. This guide explains how to customize user agent strings for HTTP requests in SpiderFoot based on the actual source code implementation in the `smicallef/spiderfoot` repository.

## Default User Agent Configuration

SpiderFoot defines its default User-Agent in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) within the global configuration dictionary:

```python
'_useragent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0'

```

This default masquerades as Firefox 62 on Windows 10 (see lines 59-60 in [[`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py)](https://github.com/smicallef/spiderfoot/blob/master/sf.py#L59-L60)).

The same file documents three supported input formats for customizing this value (lines 78-79):

1. **Direct string** — Provide any custom User-Agent text
2. **Local file with `@` prefix** — SpiderFoot randomly selects from a file containing one agent per line
3. **Remote URL** — SpiderFoot downloads a plain-text list and randomly selects from it

## How the Custom Value Propagates Through the Codebase

### Configuration Processing in sfscan.py

When a scan initializes, [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) transforms the raw configuration value into its final usable form. Line 196 invokes `optValueToData` to expand file references or fetch remote lists:

```python
self.__config['_useragent'] = self.__sf.optValueToData(self.__config['_useragent'])

```

This method handles the `@file` syntax and HTTP(S) URL resolution before any modules execute.

### Header Application in sflib.py

Every HTTP request flows through `SpiderFoot.fetchUrl` in [`sflib.py`](https://github.com/smicallef/spiderfoot/blob/main/sflib.py). Lines 558-562 implement the selection logic:

```python
if isinstance(useragent, list):
    header['User-Agent'] = random.SystemRandom().choice(useragent)
else:
    header['User-Agent'] = useragent

```

This ensures that whether you provide a single string or a list, the correct header format reaches the target server.

### Module-Level Access

All SpiderFoot modules receive the processed value via `self.opts['_useragent']` or `self.sf.opts['_useragent']`. Modules pass this directly to `fetchUrl`, meaning your customization affects **every HTTP request** across the entire scan without per-module configuration.

## Practical Customization Examples

### Set a Static Custom User Agent

Create a configuration file with your specific agent string:

```python
cat > custom_ua.json <<EOF
{
    "_useragent": "MySpiderFoot/1.0 (+https://example.com/bot)"
}
EOF

```

Execute the scan with this configuration:

```bash
python3 sf.py -s example.com -c custom_ua.json

```

### Use Random Agents from a Local File

Prepare a file with one User-Agent per line:

```bash
cat > useragents.txt <<EOF
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.0 Chrome/108.0.0.0
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Safari/605.1.15
Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0
EOF

```

Reference it with the `@` prefix:

```python
cat > random_local.json <<EOF
{
    "_useragent": "@$(pwd)/useragents.txt"
}
EOF

```

```bash
python3 sf.py -s example.com -c random_local.json

```

### Fetch Random Agents from a Remote URL

Point SpiderFoot to an external list:

```python
cat > random_remote.json <<EOF
{
    "_useragent": "https://raw.githubusercontent.com/marcusbotacin/ua-list/master/ua-list.txt"
}
EOF

```

```bash
python3 sf.py -s example.com -c random_remote.json

```

SpiderFoot downloads the list once at scan start and randomly selects for each request.

## Programmatic Override for Individual Modules

For specialized testing, override the User-Agent for a specific module instance:

```python
from spiderfoot import SpiderFoot

sf = SpiderFoot()

# Customize only the DNSDumpster module

sf.modules['sfp_dnsdumpster'].opts['_useragent'] = "CustomAgent/2.0 SecurityResearch"

sf.modules['sfp_dnsdumpster'].start()

```

This technique bypasses the global configuration for targeted testing scenarios.

## Key Files in the User Agent Implementation

| File | Location | Purpose |
|------|----------|---------|
| [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) | Lines 59-78 | Defines default value and documents the three input formats |
| [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) | Line 196 | Processes `@file` and URL references at scan initialization |
| [`sflib.py`](https://github.com/smicallef/spiderfoot/blob/main/sflib.py) | Lines 558-562 | Applies final User-Agent header in `fetchUrl` |
| Module files | Various | Inherit via `self.opts['_useragent']` for all requests |

## Summary

- **SpiderFoot's `_useragent` configuration option** controls the User-Agent header for all HTTP requests
- **Three input formats supported**: plain string, `@file` reference, or remote URL
- **Random selection** occurs via `random.SystemRandom().choice` when a list is provided
- **Global propagation** ensures consistency across every module without individual configuration
- **Source locations**: [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) for defaults, [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) for preprocessing, [`sflib.py`](https://github.com/smicallef/spiderfoot/blob/main/sflib.py) for application

## Frequently Asked Questions

### What is SpiderFoot's default User-Agent string?

SpiderFoot uses `Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:62.0) Gecko/20100101 Firefox/62.0` as defined in [`sf.py`](https://github.com/smicallef/spiderfoot/blob/main/sf.py) lines 59-60. This Firefox 62 on Windows 10 signature helps reduce detection by target servers expecting automated tools.

### How does SpiderFoot handle multiple User-Agents?

When you provide a file path prefixed with `@` or a remote URL to a text list, [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py) converts this into a Python list via `optValueToData`. Then in [`sflib.py`](https://github.com/smicallef/spiderfoot/blob/main/sflib.py) lines 558-562, `random.SystemRandom().choice(useragent)` selects a random entry for each HTTP request.

### Can different modules use different User-Agents simultaneously?

Yes, but only through programmatic manipulation. While the global configuration applies universally, you can override `self.opts['_useragent']` on individual module instances before calling `start()`. There is no built-in per-module configuration in the CLI or JSON config system.

### Where does the remote User-Agent list get downloaded?

The `optValueToData` method in [`sflib.py`](https://github.com/smicallef/spiderfoot/blob/main/sflib.py) handles URL fetching during scan initialization in [`sfscan.py`](https://github.com/smicallef/spiderfoot/blob/main/sfscan.py). The download occurs once when the scan starts, not per-request, and the resulting list remains in memory for random selection throughout the scan lifetime.