# How Holehe Achieves Concurrent Scanning of Over 120 Websites

> Discover how Holehe performs concurrent scanning of over 120 websites using async requests and Trio's structured concurrency for lightning-fast performance. Learn the technical details behind its efficiency.

- Repository: [Palenath/holehe](https://github.com/megadose/holehe)
- Tags: internals
- Published: 2026-08-23

---

**Holehe combines `httpx.AsyncClient` for non-blocking HTTP requests, Trio's structured concurrency with nurseries, and progress instrumentation to scan thousands of websites simultaneously without spawning OS threads.**

Holehe is an open-source email reconnaissance tool that checks whether a target email address is registered on hundreds of websites. To handle this massive workload efficiently, the project uses an elegant async architecture built on **Trio** and **HTTPX**. Below is a detailed breakdown of how Holehe achieves concurrent scanning, with direct references to the source code in `megadose/holehe`.

## Core Async Architecture: Three Key Components

Holehe's concurrency model rests on three tightly integrated components that work together to maximize throughput without the overhead of traditional threading.

### 1. Shared Async HTTP Client

All HTTP requests flow through a single `httpx.AsyncClient` instance created in [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py) at lines 13–14:

```python

# holehe/core.py:13-14

client = httpx.AsyncClient(timeout=timeout)

```

This **shared async client** eliminates connection overhead by reusing TCP connections across all site checks. The `timeout` parameter is user-configurable, preventing any single slow site from blocking the entire scan. Since HTTPX operates asynchronously, requests never block the event loop—enabling true parallelism without thread management.

### 2. Trio's Structured Concurrency with Nurseries

The heart of Holehe's parallel execution lies in Trio's nursery pattern, implemented in [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py) at lines 18–21:

```python

# holehe/core.py:18-21

async with trio.open_nursery() as n:
    for module in modules:
        n.start_soon(launch_module, module, email, client, out)

```

A **Trio nursery** is a context manager that supervises a group of concurrent tasks. When `n.start_soon()` is called, Trio schedules `launch_module` as a separate coroutine. The nursery guarantees that all started tasks complete before the async context exits—no orphaned requests, no resource leaks.

Each `launch_module` wrapper (defined at lines 66–71) awaits the actual site-specific function:

```python

# holehe/core.py:66-71

async def launch_module(module, email, client, out):
    try:
        await module(email, client, out)
    except Exception as e:
        pass  # failures are silently ignored to prevent one site from crashing the batch

```

This design allows **over 1,000 concurrent coroutines** (matching the number of site modules) to share the event loop cooperatively.

### 3. Live Progress Instrumentation

To provide user feedback during long-running scans, Holehe attaches `TrioProgress` to Trio's low-level instrument system at lines 16–23:

```python

# holehe/core.py:16-23

instrument = TrioProgress(len(modules))
trio.lowlevel.add_instrument(instrument)

async with trio.open_nursery() as n:
    # ... task scheduling ...

# After nursery completes

trio.lowlevel.remove_instrument(instrument)

```

The `TrioProgress` class (defined in [`holehe/instruments.py`](https://github.com/megadose/holehe/blob/main/holehe/instruments.py)) implements Trio's instrument interface to hook into task lifecycle events, rendering a live progress bar without interfering with the scanning logic.

## The Complete Scanning Workflow

Holehe's concurrent scanning follows a four-phase pipeline:

### Phase 1: Dynamic Module Discovery

At lines 36–47, `import_submodules()` recursively walks the `holehe.modules` package tree:

```python

# holehe/core.py:36-47

modules = import_submodules("holehe.modules")

```

This discovers all site-specific modules (e.g., [`holehe/modules/social_media/facebook.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/facebook.py), [`holehe/modules/shopping/amazon.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/shopping/amazon.py)) without hardcoding import statements. The modular structure makes it trivial to add new sites—just drop a Python file implementing the required async signature.

### Phase 2: Function Extraction

Lines 50–63 extract the actual callable scanning functions from loaded modules:

```python

# holehe/core.py:50-63

functions = get_functions(modules, args)

```

Each extracted function must match the signature `async def site_name(email: str, client: httpx.AsyncClient, out: list) -> None`. Functions append their findings directly to the shared `out` list, eliminating complex return value synchronization.

### Phase 3: Concurrent Execution

The nursery (lines 18–21) launches all extracted functions simultaneously. Since each site check is I/O-bound (waiting on HTTP responses), cooperative multitasking yields maximum efficiency—we're not CPU-limited, we're network-limited.

### Phase 4: Result Aggregation

Results accumulate in the thread-safe `out` list throughout execution. When the nursery context closes (all tasks finished), the client closes and the aggregated list is formatted for display.

## Why This Beats Threading or Multiprocessing

| Approach | Holehe's Method | Traditional Threading |
|----------|---------------|----------------------|
| **Memory overhead** | Single-threaded, minimal stack per coroutine (~1KB) | ~8MB stack per thread |
| **Context switching** | Cooperative (event loop decides) | Preemptive (kernel decides) |
| **Error isolation** | Nursery cancel scopes contain failures | Thread exceptions can crash process |
| **Scalability** | Thousands of concurrent connections | Hundreds before OS limits |

Trio's structured concurrency provides **automatic cleanup guarantees**: if any site check raises an unhandled exception, the nursery cancels all sibling tasks and propagates the error—preventing resource leaks that plague callback-based async code.

## Code Pattern Used in Site Modules

Every site module follows this async pattern, as seen in [`holehe/modules/social_media/facebook.py`](https://github.com/megadose/holehe/blob/main/holehe/modules/social_media/facebook.py):

```python

# Typical site module implementation

import httpx

async def facebook(email: str, client: httpx.AsyncClient, out: list) -> None:
    data = {"email": email}
    try:
        r = await client.post(
            "https://www.facebook.com/login/identify",
            data=data,
            headers={"User-Agent": "..."}
        )
        # Parse response, detect account existence

        exists = "account" in r.text.lower()
        out.append({
            "name": "facebook",
            "exists": exists,
            "email_recovery": None,
            "phoneNumber": None,
            "others": None
        })
    except Exception:
        # Failures are caught; other sites continue unaffected

        pass

```

The shared `client` parameter ensures connection pooling, while direct list mutation bypasses need for async queues or locks.

## Summary

- **HTTPX async client** (`holehe/core.py:13–14`) provides non-blocking, connection-pooled HTTP requests.
- **Trio nurseries** (`holehe/core.py:18–21`) manage thousands of concurrent coroutines with automatic lifecycle management.
- **Progress instrumentation** (`holehe/core.py:16–23`) adds user visibility without blocking logic.
- **Dynamic module loading** (`holehe/core.py:36–47`) scales the site list without code changes.
- **Structured concurrency** guarantees cleanup and error isolation impossible with raw threading.

This architecture lets Holehe query **over 120 websites in seconds**—and the code scales linearly as module count grows.

## Frequently Asked Questions

### What is the maximum number of websites Holehe can scan concurrently?

The theoretical limit depends on available file descriptors and memory, not artificial code constraints. In practice, Holehe handles **1,000+ sites** simultaneously on modest hardware. Trio coroutines use roughly 1KB of memory each versus 8MB+ for OS threads, so the bottleneck becomes network bandwidth and remote server response times, not local resources.

### Why does Holehe use Trio instead of asyncio?

Trio provides **structured concurrency**—guarantees that tasks cannot outlive their parent context. This eliminates a class of resource leaks common in `asyncio` where dangling tasks continue after program exit. Trio's nursery model also offers superior error propagation: one failing site can optionally cancel siblings or be isolated, with explicit control unavailable in `asyncio`'s loosely coupled task creation.

### How does the shared HTTP client improve performance?

`httpx.AsyncClient` maintains a **connection pool** underneath. When 500 site modules request different endpoints, HTTPX reuses established TCP connections where possible (HTTP/1.1 keep-alive or HTTP/2 multiplexing). Without this, each request would pay TCP handshake and TLS negotiation overhead—adding seconds per site. The shared client also respects global rate limits and timeout settings consistently.

### What happens if a single website scan fails?

Failure isolation is built into `launch_module` (`holehe/core.py:66–71`). The wrapper catches all exceptions silently, ensuring one slow or broken site cannot crash the entire batch. The scan continues for remaining sites, and partial results are still returned. This **defensive design** is essential when scanning unreliable third-party endpoints with varying uptime.