How NSFW Site Filtering Works in Sherlock: A Technical Deep Dive

Sherlock's NSFW site filtering works by reading an optional isNSFW boolean flag from each site's metadata in the JSON manifest, storing it in the SiteInformation.is_nsfw attribute, and conditionally removing those sites via the remove_nsfw_sites() method unless the user provides the --nsfw command-line flag.

The Sherlock project is a command-line username enumeration tool that searches across hundreds of social platforms. To ensure safe-for-work results by default, the codebase implements a robust NSFW filtering mechanism that automatically excludes adult content platforms from scans unless explicitly requested.

Site Metadata and the isNSFW Flag

Sherlock loads site definitions from a JSON manifest (typically data.json) where each entry may contain an optional isNSFW key. When the SitesInformation class instantiates, it iterates through the manifest and maps the JSON field to the is_nsfw attribute of each SiteInformation object, defaulting to False if the key is absent.

In sherlock_project/sites.py, the initialization logic captures this flag:

self.sites[site_name] = SiteInformation(
    site_name,
    site_data[site_name]["urlMain"],
    site_data[site_name]["url"],
    site_data[site_name]["username_claimed"],
    site_data[site_name],
    site_data[site_name].get("isNSFW", False)   # ← NSFW flag

)

Source: sherlock_project/sites.py (lines 94-102)

The --nsfw Command-Line Flag

The filtering behavior is controlled by a boolean argument defined in the CLI parser. By default, this flag is False, meaning NSFW sites are excluded from the scan set.

In sherlock_project/sherlock.py, the argument is defined as:

parser.add_argument(
    "--nsfw",
    action="store_true",
    default=False,
    help="Include checking of NSFW sites from default list.",
)

Source: sherlock_project/sherlock.py (lines 72-76)

Conditional Removal in the Main Execution Flow

After parsing arguments, Sherlock checks whether the --nsfw flag was provided. If the flag is absent, the code immediately invokes the removal method to filter the site list before processing begins.

if not args.nsfw:
    sites.remove_nsfw_sites(do_not_remove=args.site_list)

Source: sherlock_project/sherlock.py (lines 79-81)

The remove_nsfw_sites() Filtering Implementation

The actual filtering logic resides in the SitesInformation.remove_nsfw_sites() method in sherlock_project/sites.py. This method reconstructs the internal sites dictionary, skipping any entry where is_nsfw is True unless the site name appears in an explicit whitelist.

def remove_nsfw_sites(self, do_not_remove: list = []):
    sites = {}
    do_not_remove = [site.casefold() for site in do_not_remove]
    for site in self.sites:
        if self.sites[site].is_nsfw and site.casefold() not in do_not_remove:
            continue                     # ← skip NSFW sites

        sites[site] = self.sites[site]   # keep everything else

    self.sites = sites

The method uses casefold() for case-insensitive comparison, ensuring that site names like "Pornhub" and "pornhub" are treated identically. The do_not_remove parameter supports the --site whitelist functionality, allowing specific NSFW platforms to remain in the scan set even when the global filter is active.

Source: sherlock_project/sites.py (lines 13-30)

Handling the --site Whitelist Exception

When users specify particular sites via the --site argument, those site names are passed to remove_nsfw_sites() as the do_not_remove parameter. This architecture allows you to explicitly request an NSFW site while maintaining the filter for all other adult platforms.

For example, if you run:

sherlock --site Pornhub user123

The remove_nsfw_sites() method receives do_not_remove=['Pornhub'], causing it to skip the removal logic for that specific site while still filtering out other NSFW entries.

Practical Usage Examples

Default Behavior (NSFW Sites Excluded)

Run Sherlock without flags to automatically omit adult platforms:

$ sherlock user123

# NSFW sites such as "Pornhub" are automatically omitted.

Include All NSFW Sites

Add the --nsfw flag to bypass the removal step entirely:

$ sherlock --nsfw user123

# All sites from the manifest, including those with "isNSFW": true, are checked.

Programmatic Usage

You can access the same filtering logic when using Sherlock as a library:

from sherlock_project.sites import SitesInformation

# Load the full site list

sites = SitesInformation()

# By default, NSFW sites are present

print('Total sites:', len(sites))

# Remove NSFW sites (same logic as the CLI when --nsfw is omitted)

sites.remove_nsfw_sites()

print('After filtering:', len(sites))

Explicit NSFW Site Selection

Target a specific NSFW platform while keeping the filter active for others:

$ sherlock --site Pornhub user123

# The `--site` argument adds "Pornhub" to the whitelist.

# Sherlock will call remove_nsfw_sites(do_not_remove=['Pornhub'])

Summary

• Site metadata in the JSON manifest uses the isNSFW key to mark adult content platforms
• Data model: SiteInformation stores this state in the is_nsfw attribute during instantiation
• CLI control: The --nsfw flag defaults to False, triggering automatic removal of flagged sites
• Filter implementation: remove_nsfw_sites() in sherlock_project/sites.py reconstructs the site dictionary, excluding NSFW entries unless whitelisted
• Whitelist support: The --site argument passes specific site names to the filter, allowing selective inclusion of NSFW platforms
• Core files: Filtering logic resides in sherlock_project/sites.py, while the CLI trigger is in sherlock_project/sherlock.py

Frequently Asked Questions

How does Sherlock determine which sites are NSFW?

According to the Sherlock source code, the determination comes from the isNSFW boolean field in the site manifest file (data.json). When SitesInformation loads the data, it extracts this value using site_data[site_name].get("isNSFW", False) and assigns it to the is_nsfw attribute of each SiteInformation instance, defaulting to False if the key is absent.

Can I search a specific NSFW site without enabling --nsfw for all sites?

Yes. When you use the --site argument to specify particular platforms, those site names are passed to remove_nsfw_sites() as the do_not_remove whitelist parameter. This allows you to check a specific adult platform while keeping the NSFW filter active for all other sites, ensuring you only access the requested NSFW content.

Where is the NSFW filtering logic implemented in the codebase?

The core filtering logic is implemented in the remove_nsfw_sites() method within sherlock_project/sites.py (lines 13-30). The CLI argument handling that triggers this filtering is located in sherlock_project/sherlock.py (lines 72-81), where the code checks the args.nsfw boolean and conditionally invokes the removal method.

Is NSFW filtering enabled by default in Sherlock?

Yes. The --nsfw argument defaults to False in the argument parser configuration defined in sherlock_project/sherlock.py. This means remove_nsfw_sites() is automatically called during initialization unless you explicitly add the --nsfw flag to your command, ensuring all searches are safe-for-work by default.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}