# Sherlock Custom HTTP Headers: Default User-Agent and Per-Site Overrides

> Discover how Sherlock uses a default User-Agent and allows custom HTTP headers per site via JSON configuration. Override or append headers for tailored requests.

- Repository: [Sherlock/sherlock](https://github.com/sherlock-project/sherlock)
- Tags: internals
- Published: 2026-03-02

---

**Sherlock sends a Firefox User-Agent header with every request, but you can override or append custom HTTP headers per site by defining a `headers` object in that site's JSON configuration.**

The sherlock-project/sherlock tool queries hundreds of websites to check username availability, relying on specific HTTP headers to ensure reliable responses. Understanding how to configure custom headers in Sherlock allows you to handle sites with strict bot protection, API authentication requirements, or non-standard content type expectations.

## Default User-Agent Header in Sherlock

In [`sherlock_project/sherlock.py`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/sherlock.py), Sherlock constructs a default headers dictionary that is applied to every network request:

```python

# sherlock_project/sherlock.py

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0",
}

```

This **User-Agent** mimics a recent Firefox browser on Linux to minimize blocking by simple bot detection filters. The dictionary is defined before the site-specific loop and serves as the base for all outgoing probes.

## Per-Site Header Overrides in data.json

For sites requiring non-standard headers, Sherlock supports per-site customization through the **manifest file** at [`sherlock_project/resources/data.json`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/resources/data.json). Each site entry can include a `headers` object containing key-value pairs that merge with the defaults.

The merging logic in [`sherlock_project/sherlock.py`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/sherlock.py) uses Python's `dict.update()` method:

```python

# sherlock_project/sherlock.py

if "headers" in net_info:
    # Override/append any extra headers required by a given site.

    headers.update(net_info["headers"])

```

When `headers` exists in the site's configuration, those values **override** matching keys in the default dictionary (e.g., replacing the User-Agent) or **append** entirely new headers (e.g., `Accept`, `Content-Type`, or authentication tokens).

### Examples from the Sherlock Manifest

Several sites in the default [`data.json`](https://github.com/sherlock-project/sherlock/blob/main/data.json) utilize custom headers:

- **Discord**: Sets `"Content-Type": "application/json"` for POST requests to the username availability endpoint (around line 725)
- **Kongregate**: Specifies `"Accept": "text/html"` to ensure proper content negotiation (around line 1395)

These entries demonstrate how the community maintains site-specific requirements without modifying core code.

## How to Override Headers Per Site

You can customize headers for any site using two methods: editing the built-in manifest or supplying an external site file.

### Method 1: Modify resources/data.json

Edit [`sherlock_project/resources/data.json`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/resources/data.json) and add a `headers` object to the target site entry:

```json
"ExampleSite": {
    "errorType": "status_code",
    "url": "https://www.examplesite.com/{}",
    "urlMain": "https://www.examplesite.com/",
    "username_claimed": "blue",
    "headers": {
        "User-Agent": "MyCustomAgent/2.0",
        "Accept": "application/json",
        "X-Custom-Header": "value"
    }
}

```

Save the file and run Sherlock normally. The next request to ExampleSite will include your custom User-Agent and additional headers.

### Method 2: Use a Custom Site File

Create a separate JSON file (e.g., [`custom_sites.json`](https://github.com/sherlock-project/sherlock/blob/main/custom_sites.json)) containing only the sites you want to modify:

```json
{
    "MyApiSite": {
        "errorType": "status_code",
        "url": "https://api.mysite.com/users/{}",
        "urlMain": "https://mysite.com/",
        "username_claimed": "admin",
        "headers": {
            "User-Agent": "SherlockBot/1.0",
            "Authorization": "Bearer token123"
        }
    }
}

```

Launch Sherlock with the `--site-file` argument to load your custom definitions:

```bash
sherlock targetusername --site-file custom_sites.json

```

Sherlock merges your custom headers with the defaults for any site defined in your file, overriding the built-in list completely for those entries.

## Practical Code Examples

The following examples demonstrate how Sherlock constructs requests with different header configurations.

### Default Request Behavior

Running Sherlock without custom site definitions sends only the default Firefox User-Agent:

```bash
sherlock user123

```

Each probe includes:

```http
GET https://targetsite.com/user123
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0

```

### Site with Custom Content-Type (Discord)

When querying Discord, Sherlock automatically includes the JSON content type header defined in [`data.json`](https://github.com/sherlock-project/sherlock/blob/main/data.json):

```bash
sherlock user123 --site Discord

```

The resulting POST request contains:

```http
POST https://discord.com/api/v9/unique-username/username-attempt-unauthed
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0
Content-Type: application/json

```

### Custom API Key Header

Using the custom site file method shown earlier, you can inject authentication headers:

```bash
sherlock user123 --site-file my_sites.json

```

For the defined site, the request includes:

```http
GET https://api.mysite.com/users/user123
User-Agent: SherlockBot/1.0
Authorization: Bearer token123

```

## Summary

- Sherlock uses a default **User-Agent** header (`Mozilla/5.0...Firefox/129.0`) for all requests defined in [`sherlock_project/sherlock.py`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/sherlock.py)
- Per-site overrides are configured via the `headers` object in [`resources/data.json`](https://github.com/sherlock-project/sherlock/blob/main/resources/data.json)
- The code merges custom headers using `headers.update(net_info["headers"])`, allowing both overrides and additions
- You can supply alternate header configurations using the `--site-file` runtime argument without modifying core source files

## Frequently Asked Questions

### What is the default User-Agent string used by Sherlock?

Sherlock identifies as `Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0` for every request unless overridden per site. This string is hardcoded in [`sherlock_project/sherlock.py`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/sherlock.py) as the base headers dictionary.

### Can I override headers for just one specific site without editing the main data.json?

Yes. Create a custom JSON file containing only that site's definition with your desired `headers` object, then run Sherlock with `--site-file yourfile.json`. This loads your configuration instead of the built-in entry for that site.

### Does Sherlock support authentication headers like Authorization or API keys?

Absolutely. Any header key-value pair defined in a site's `headers` object is passed directly to the request. You can include `Authorization`, `X-API-Key`, or custom authentication tokens required by the target platform.

### How do I check if my custom headers are being sent correctly?

Use Sherlock's verbose output options or inspect network traffic with a proxy tool like Burp Suite or Wireshark. The headers defined in [`data.json`](https://github.com/sherlock-project/sherlock/blob/main/data.json) or your custom site file are merged via `headers.update()` in [`sherlock_project/sherlock.py`](https://github.com/sherlock-project/sherlock/blob/main/sherlock_project/sherlock.py) and sent with every request to that specific site.