How Holehe Uses BeautifulSoup for Parsing Website Responses in Email Reconnaissance

Holehe treats every target site as a small web-scraping problem, using BeautifulSoup to extract authentication tokens, CSRF values, and hidden form fields from raw HTML responses.

Holehe is an open-source reconnaissance tool by @megadose that checks whether an email address is registered on hundreds of websites. A critical component of this process is parsing website responses with BeautifulSoup to locate dynamic data embedded in HTML. This article examines how Holehe implements HTML parsing across its modular architecture, with concrete examples from the source code.

The BeautifulSoup Integration Pattern

Holehe follows a consistent four-step workflow for every service module. The BeautifulSoup library is imported once in the core module and instantiated as needed throughout individual site handlers.

Core Import Location

In holehe/core.py, the import establishes the foundation:

from bs4 import BeautifulSoup

This single import serves all modules. The actual instantiation happens within each site-specific function, keeping memory usage minimal and localized.

Standard Parsing Flow

Every module implements this pattern:

1. Fetch — Make an asynchronous HTTP request via httpx.AsyncClient
2. Parse — Feed response.text to BeautifulSoup(response.text, "html.parser")
3. Navigate — Use BeautifulSoup's tree-searching methods (find_all, find, text filtering)
4. Extract — Pull the specific token, ID, or hidden value needed for the next request

The heavy lifting of network I/O and concurrency is handled by httpx and trio, while BeautifulSoup focuses solely on lightweight HTML extraction.

Concrete Implementation: The Tumblr Module

The Tumblr handler in holehe/modules/social_media/tumblr.py (lines 14–27) demonstrates the complete parsing workflow:

# 1. Initial GET request to fetch the landing page

response = await client.get("https://www.tumblr.com/", headers={...})

# 2. Parse HTML with BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")

# 3. Locate script tags containing the API token

script_tags = soup.find_all("script")
bearer = [
    ";".join(tag.text.split("window['___INITIAL_STATE___'] = ")[-1].split(";")[:-1])
    .split('{"API_TOKEN":"')[-1]
    .split('","extraHeaders":"{}"}')[0]
    for tag in script_tags
    if "window['___INITIAL_STATE___']" in tag.text
][0]

# 4. Use extracted token in subsequent API request

auth_header = f"Bearer {bearer}"
csrf_resp = await client.get(
    "https://www.tumblr.com/api/v2/radar?...",
    headers={"Authorization": auth_header, ...}
)

This pattern—GET → parse → token extraction → authenticated request—is replicated across dozens of modules. The parsing logic varies based on how each site structures its HTML, but the architectural approach remains constant.

Parsing Patterns Across Different Services

Holehe adapts BeautifulSoup extraction to diverse HTML structures. Here are representative implementations from various service categories:

Hidden Form Fields: Odnoklassniki

The Russian social network Odnoklassniki stores its authenticity_token in a hidden form field. In holehe/modules/social_media/odnoklassniki.py (lines 26–28):

data = {
    'authenticity_token': soup.find('input', {'name': 'authenticity_token'})['value'],
    ...
}

Session Identifiers: Amazon

The Amazon module in holehe/modules/shopping/amazon.py (lines 15–20) extracts multiple hidden values from the login page:

data = {
    'session-id': soup.find('input', {'name': 'session-id'})['value'],
    ...
}

Script Tag Parsing: Redtube and Codecademy

Some services embed tokens in <script> tags rather than standard HTML elements.

Redtube (holehe/modules/porn/redtube.py, lines 22–24):

script_tags = soup.find_all("script")

# Extract video ID from inline JavaScript

Codecademy (holehe/modules/programing/codecademy.py, lines 25–27):

# Pull CSRF token from script block for registration request

Reusable Parsing Architecture

Holehe's modular design allows code reuse without sacrificing flexibility. A generic helper pattern emerges:

async def fetch_and_parse(url: str, client: httpx.AsyncClient, headers: dict = None):
    resp = await client.get(url, headers=headers or {})
    resp.raise_for_status()
    return BeautifulSoup(resp.text, "html.parser")

Modules invoke this pattern then apply site-specific extraction logic. For Instagram-like services that embed JSON in script tags:

async def instagram(email, client, out):
    soup = await fetch_and_parse(
        "https://www.instagram.com/accounts/email_sent/", 
        client
    )
    json_blob = next(
        tag.text for tag in soup.find_all("script")
        if "window._sharedData" in tag.text
    )
    # Parse JSON blob and determine email registration status

Key Source Files and Their Roles

File	Purpose	Parser Usage
holehe/core.py	Entry point, shared imports	from bs4 import BeautifulSoup
holehe/modules/social_media/tumblr.py	Tumblr email check	Full flow: GET → parse → token → API call
holehe/modules/social_media/odnoklassniki.py	Odnoklassniki check	Hidden form field extraction
holehe/modules/shopping/amazon.py	Amazon check	Session ID from landing page
holehe/modules/porn/redtube.py	Redtube check	Script tag traversal for video ID
holehe/modules/programing/codecademy.py	Codecademy check	CSRF token from inline JavaScript

Design Advantages

Holehe's BeautifulSoup implementation prioritizes minimal, targeted extraction over comprehensive scraping:

• Performance: Only parses what each specific request requires
• Maintainability: Site-specific parsers are isolated in dedicated modules
• Resilience: Small HTML changes can be fixed module-by-module without core rewrites
• Scalability: New services add new parser functions without architectural changes

The html.parser backend (Python's built-in parser) avoids external dependencies like lxml while remaining sufficient for Holehe's extraction needs.

Summary

• BeautifulSoup is imported once in holehe/core.py and instantiated per-module as BeautifulSoup(response.text, "html.parser")
• Every service module follows fetch → parse → extract → continue workflow
• Tumblr (tumblr.py lines 14–27) exemplifies the full pattern: landing page → API token extraction → authenticated secondary request
• Extraction targets vary: hidden form fields (Odnoklassniki, Amazon), script tags (Redtube, Codecademy), or inline JavaScript variables
• Architecture separates concerns: httpx handles async I/O, BeautifulSoup handles HTML navigation, site modules handle service-specific logic

Frequently Asked Questions

Why does Holehe use BeautifulSoup instead of regex for HTML parsing?

BeautifulSoup's tree-structured navigation provides reliable element selection that regex cannot guarantee. Holehe needs precise extraction of attributes like value from specific <input> tags or text from <script> blocks with particular content markers—operations that are fragile with regex but robust with DOM parsing. The html.parser backend adds no external dependencies.

Does Holehe cache or reuse BeautifulSoup objects across requests?

No. Each site module instantiates BeautifulSoup fresh for every HTTP response. This prevents state pollution between services and keeps memory usage bounded. The pattern soup = BeautifulSoup(response.text, "html.parser") appears in each async function independently.

What parser backend does Holehe specify?

Holehe consistently uses "html.parser" (Python's built-in html.parser module). This avoids requiring lxml or html5lib installations while providing sufficient performance for single-page extractions. The source shows no conditional parser selection or fallback logic.

How does Holehe handle sites that load content dynamically with JavaScript?

Holehe's BeautifulSoup parsing operates on initial HTTP responses only. For JavaScript-heavy applications like modern React or Angular apps, Holehe extracts pre-rendered data from script tags (as seen in Tumblr's window['___INITIAL_STATE___'] approach) or makes direct API calls discovered through static analysis. The tool does not execute JavaScript itself.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}