Sherlock Custom HTTP Headers: Default User-Agent and Per-Site Overrides

Sherlock sends a Firefox User-Agent header with every request, but you can override or append custom HTTP headers per site by defining a headers object in that site's JSON configuration.

The sherlock-project/sherlock tool queries hundreds of websites to check username availability, relying on specific HTTP headers to ensure reliable responses. Understanding how to configure custom headers in Sherlock allows you to handle sites with strict bot protection, API authentication requirements, or non-standard content type expectations.

Default User-Agent Header in Sherlock

In sherlock_project/sherlock.py, Sherlock constructs a default headers dictionary that is applied to every network request:

# sherlock_project/sherlock.py

headers = {
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0",
}

This User-Agent mimics a recent Firefox browser on Linux to minimize blocking by simple bot detection filters. The dictionary is defined before the site-specific loop and serves as the base for all outgoing probes.

Per-Site Header Overrides in data.json

For sites requiring non-standard headers, Sherlock supports per-site customization through the manifest file at sherlock_project/resources/data.json. Each site entry can include a headers object containing key-value pairs that merge with the defaults.

The merging logic in sherlock_project/sherlock.py uses Python's dict.update() method:

# sherlock_project/sherlock.py

if "headers" in net_info:
    # Override/append any extra headers required by a given site.

    headers.update(net_info["headers"])

When headers exists in the site's configuration, those values override matching keys in the default dictionary (e.g., replacing the User-Agent) or append entirely new headers (e.g., Accept, Content-Type, or authentication tokens).

Examples from the Sherlock Manifest

Several sites in the default data.json utilize custom headers:

• Discord: Sets "Content-Type": "application/json" for POST requests to the username availability endpoint (around line 725)
• Kongregate: Specifies "Accept": "text/html" to ensure proper content negotiation (around line 1395)

These entries demonstrate how the community maintains site-specific requirements without modifying core code.

How to Override Headers Per Site

You can customize headers for any site using two methods: editing the built-in manifest or supplying an external site file.

Method 1: Modify resources/data.json

Edit sherlock_project/resources/data.json and add a headers object to the target site entry:

"ExampleSite": {
    "errorType": "status_code",
    "url": "https://www.examplesite.com/{}",
    "urlMain": "https://www.examplesite.com/",
    "username_claimed": "blue",
    "headers": {
        "User-Agent": "MyCustomAgent/2.0",
        "Accept": "application/json",
        "X-Custom-Header": "value"
    }
}

Save the file and run Sherlock normally. The next request to ExampleSite will include your custom User-Agent and additional headers.

Method 2: Use a Custom Site File

Create a separate JSON file (e.g., custom_sites.json) containing only the sites you want to modify:

{
    "MyApiSite": {
        "errorType": "status_code",
        "url": "https://api.mysite.com/users/{}",
        "urlMain": "https://mysite.com/",
        "username_claimed": "admin",
        "headers": {
            "User-Agent": "SherlockBot/1.0",
            "Authorization": "Bearer token123"
        }
    }
}

Launch Sherlock with the --site-file argument to load your custom definitions:

sherlock targetusername --site-file custom_sites.json

Sherlock merges your custom headers with the defaults for any site defined in your file, overriding the built-in list completely for those entries.

Practical Code Examples

The following examples demonstrate how Sherlock constructs requests with different header configurations.

Default Request Behavior

Running Sherlock without custom site definitions sends only the default Firefox User-Agent:

sherlock user123

Each probe includes:

GET https://targetsite.com/user123
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0

Site with Custom Content-Type (Discord)

When querying Discord, Sherlock automatically includes the JSON content type header defined in data.json:

sherlock user123 --site Discord

The resulting POST request contains:

POST https://discord.com/api/v9/unique-username/username-attempt-unauthed
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0
Content-Type: application/json

Custom API Key Header

Using the custom site file method shown earlier, you can inject authentication headers:

sherlock user123 --site-file my_sites.json

For the defined site, the request includes:

GET https://api.mysite.com/users/user123
User-Agent: SherlockBot/1.0
Authorization: Bearer token123

Summary

• Sherlock uses a default User-Agent header (Mozilla/5.0...Firefox/129.0) for all requests defined in sherlock_project/sherlock.py
• Per-site overrides are configured via the headers object in resources/data.json
• The code merges custom headers using headers.update(net_info["headers"]), allowing both overrides and additions
• You can supply alternate header configurations using the --site-file runtime argument without modifying core source files

Frequently Asked Questions

What is the default User-Agent string used by Sherlock?

Sherlock identifies as Mozilla/5.0 (X11; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0 for every request unless overridden per site. This string is hardcoded in sherlock_project/sherlock.py as the base headers dictionary.

Can I override headers for just one specific site without editing the main data.json?

Yes. Create a custom JSON file containing only that site's definition with your desired headers object, then run Sherlock with --site-file yourfile.json. This loads your configuration instead of the built-in entry for that site.

Does Sherlock support authentication headers like Authorization or API keys?

Absolutely. Any header key-value pair defined in a site's headers object is passed directly to the request. You can include Authorization, X-API-Key, or custom authentication tokens required by the target platform.

How do I check if my custom headers are being sent correctly?

Use Sherlock's verbose output options or inspect network traffic with a proxy tool like Burp Suite or Wireshark. The headers defined in data.json or your custom site file are merged via headers.update() in sherlock_project/sherlock.py and sent with every request to that specific site.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}