# How Holehe Uses BeautifulSoup for Parsing Website Responses in Email Reconnaissance

> Learn how Holehe uses BeautifulSoup to parse website responses for email reconnaissance. Extract authentication tokens, CSRF values, and form fields from HTML.

- Repository: [Palenath/holehe](https://github.com/megadose/holehe)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Holehe treats every target site as a small web-scraping problem, using BeautifulSoup to extract authentication tokens, CSRF values, and hidden form fields from raw HTML responses.**

Holehe is an open-source reconnaissance tool by @megadose that checks whether an email address is registered on hundreds of websites. A critical component of this process is **parsing website responses with BeautifulSoup** to locate dynamic data embedded in HTML. This article examines how Holehe implements HTML parsing across its modular architecture, with concrete examples from the source code.

## The BeautifulSoup Integration Pattern

Holehe follows a consistent four-step workflow for every service module. The BeautifulSoup library is imported once in the core module and instantiated as needed throughout individual site handlers.

### Core Import Location

In [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py), the import establishes the foundation:

```python
from bs4 import BeautifulSoup

```

This single import serves all modules. The actual instantiation happens within each site-specific function, keeping memory usage minimal and localized.

### Standard Parsing Flow

Every module implements this pattern:

1. **Fetch** — Make an asynchronous HTTP request via `httpx.AsyncClient`
2. **Parse** — Feed `response.text` to `BeautifulSoup(response.text, "html.parser")`
3. **Navigate** — Use BeautifulSoup's tree-searching methods (`find_all`, `find`, text filtering)
4. **Extract** — Pull the specific token, ID, or hidden value needed for the next request

The heavy lifting of network I/O and concurrency is handled by `httpx` and `trio`, while BeautifulSoup focuses solely on lightweight HTML extraction.

## Concrete Implementation: The Tumblr Module

The Tumblr handler in [`holehe/modules/social_media/tumblr.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/tumblr.py) (lines 14–27) demonstrates the complete parsing workflow:

```python

# 1. Initial GET request to fetch the landing page

response = await client.get("https://www.tumblr.com/", headers={...})

# 2. Parse HTML with BeautifulSoup

soup = BeautifulSoup(response.text, "html.parser")

# 3. Locate script tags containing the API token

script_tags = soup.find_all("script")
bearer = [
    ";".join(tag.text.split("window['___INITIAL_STATE___'] = ")[-1].split(";")[:-1])
    .split('{"API_TOKEN":"')[-1]
    .split('","extraHeaders":"{}"}')[0]
    for tag in script_tags
    if "window['___INITIAL_STATE___']" in tag.text
][0]

# 4. Use extracted token in subsequent API request

auth_header = f"Bearer {bearer}"
csrf_resp = await client.get(
    "https://www.tumblr.com/api/v2/radar?...",
    headers={"Authorization": auth_header, ...}
)

```

This pattern—**GET → parse → token extraction → authenticated request**—is replicated across dozens of modules. The parsing logic varies based on how each site structures its HTML, but the architectural approach remains constant.

## Parsing Patterns Across Different Services

Holehe adapts BeautifulSoup extraction to diverse HTML structures. Here are representative implementations from various service categories:

### Hidden Form Fields: Odnoklassniki

The Russian social network Odnoklassniki stores its `authenticity_token` in a hidden form field. In [`holehe/modules/social_media/odnoklassniki.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/odnoklassniki.py) (lines 26–28):

```python
data = {
    'authenticity_token': soup.find('input', {'name': 'authenticity_token'})['value'],
    ...
}

```

### Session Identifiers: Amazon

The Amazon module in [`holehe/modules/shopping/amazon.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/shopping/amazon.py) (lines 15–20) extracts multiple hidden values from the login page:

```python
data = {
    'session-id': soup.find('input', {'name': 'session-id'})['value'],
    ...
}

```

### Script Tag Parsing: Redtube and Codecademy

Some services embed tokens in `<script>` tags rather than standard HTML elements.

**Redtube** ([`holehe/modules/porn/redtube.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/porn/redtube.py), lines 22–24):

```python
script_tags = soup.find_all("script")

# Extract video ID from inline JavaScript

```

**Codecademy** ([`holehe/modules/programing/codecademy.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/programing/codecademy.py), lines 25–27):

```python

# Pull CSRF token from script block for registration request

```

## Reusable Parsing Architecture

Holehe's modular design allows code reuse without sacrificing flexibility. A generic helper pattern emerges:

```python
async def fetch_and_parse(url: str, client: httpx.AsyncClient, headers: dict = None):
    resp = await client.get(url, headers=headers or {})
    resp.raise_for_status()
    return BeautifulSoup(resp.text, "html.parser")

```

Modules invoke this pattern then apply site-specific extraction logic. For Instagram-like services that embed JSON in script tags:

```python
async def instagram(email, client, out):
    soup = await fetch_and_parse(
        "https://www.instagram.com/accounts/email_sent/", 
        client
    )
    json_blob = next(
        tag.text for tag in soup.find_all("script")
        if "window._sharedData" in tag.text
    )
    # Parse JSON blob and determine email registration status

```

## Key Source Files and Their Roles

| File | Purpose | Parser Usage |
|------|---------|-------------|
| [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py) | Entry point, shared imports | `from bs4 import BeautifulSoup` |
| [`holehe/modules/social_media/tumblr.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/tumblr.py) | Tumblr email check | Full flow: GET → parse → token → API call |
| [`holehe/modules/social_media/odnoklassniki.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/odnoklassniki.py) | Odnoklassniki check | Hidden form field extraction |
| [`holehe/modules/shopping/amazon.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/shopping/amazon.py) | Amazon check | Session ID from landing page |
| [`holehe/modules/porn/redtube.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/porn/redtube.py) | Redtube check | Script tag traversal for video ID |
| [`holehe/modules/programing/codecademy.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/programing/codecademy.py) | Codecademy check | CSRF token from inline JavaScript |

## Design Advantages

Holehe's BeautifulSoup implementation prioritizes **minimal, targeted extraction** over comprehensive scraping:

- **Performance**: Only parses what each specific request requires
- **Maintainability**: Site-specific parsers are isolated in dedicated modules
- **Resilience**: Small HTML changes can be fixed module-by-module without core rewrites
- **Scalability**: New services add new parser functions without architectural changes

The `html.parser` backend (Python's built-in parser) avoids external dependencies like `lxml` while remaining sufficient for Holehe's extraction needs.

## Summary

- **BeautifulSoup** is imported once in [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py) and instantiated per-module as `BeautifulSoup(response.text, "html.parser")`
- **Every service module** follows fetch → parse → extract → continue workflow
- **Tumblr** ([`tumblr.py`](https://github.com/megadose/holehe/blob/main/tumblr.py) lines 14–27) exemplifies the full pattern: landing page → API token extraction → authenticated secondary request
- **Extraction targets vary**: hidden form fields (Odnoklassniki, Amazon), script tags (Redtube, Codecademy), or inline JavaScript variables
- **Architecture separates concerns**: `httpx` handles async I/O, BeautifulSoup handles HTML navigation, site modules handle service-specific logic

## Frequently Asked Questions

### Why does Holehe use BeautifulSoup instead of regex for HTML parsing?

BeautifulSoup's tree-structured navigation provides reliable element selection that regex cannot guarantee. Holehe needs precise extraction of attributes like `value` from specific `<input>` tags or text from `<script>` blocks with particular content markers—operations that are fragile with regex but robust with DOM parsing. The `html.parser` backend adds no external dependencies.

### Does Holehe cache or reuse BeautifulSoup objects across requests?

No. Each site module instantiates `BeautifulSoup` fresh for every HTTP response. This prevents state pollution between services and keeps memory usage bounded. The pattern `soup = BeautifulSoup(response.text, "html.parser")` appears in each async function independently.

### What parser backend does Holehe specify?

Holehe consistently uses `"html.parser"` (Python's built-in `html.parser` module). This avoids requiring `lxml` or `html5lib` installations while providing sufficient performance for single-page extractions. The source shows no conditional parser selection or fallback logic.

### How does Holehe handle sites that load content dynamically with JavaScript?

Holehe's BeautifulSoup parsing operates on **initial HTTP responses only**. For JavaScript-heavy applications like modern React or Angular apps, Holehe extracts pre-rendered data from script tags (as seen in Tumblr's `window['___INITIAL_STATE___']` approach) or makes direct API calls discovered through static analysis. The tool does not execute JavaScript itself.