Performance Considerations for Concurrently Checking 120+ Websites with Holehe

Holehe achieves high-throughput email verification across hundreds of sites by using Trio's structured concurrency with a shared HTTPX async client, but optimal performance requires tuning connection limits, timeouts, and module selection.

When scanning an email address against 120 or more websites, understanding Holehe's concurrency model is essential for maximizing speed without triggering rate limits. This guide examines the architectural decisions in megadose/holehe that affect performance and provides actionable optimizations based on the source code implementation.

How Holehe Handles Concurrent Requests

Holehe's concurrency engine resides in holehe/core.py. The tool orchestrates all site checks through Trio nurseries and a shared asynchronous HTTP client.

The Async Client Architecture

In core.py:L13-L14, Holehe instantiates a single httpx.AsyncClient:

client = httpx.AsyncClient(timeout=timeout)

This single client is passed to every module, enabling connection reuse across all 120+ site checks. The HTTPX default limits (approximately 100 concurrent connections) cap how many simultaneous TCP handshakes occur.

To increase throughput in a custom fork, override these defaults:

import httpx

limits = httpx.Limits(
    max_keepalive_connections=200,
    max_connections=200
)
client = httpx.AsyncClient(timeout=10, limits=limits)

Trio Nursery Task Dispatch

The parallel execution happens in core.py:L18-L21:

async with trio.open_nursery() as n:
    for function in get_functions(modules, args):
        n.start_soon(launch_module, function, args.email, client, out)

Every module starts immediately as a separate coroutine. With 120+ websites, this creates 120+ concurrent tasks competing for the event loop's attention and the HTTP connection pool.

Key Bottlenecks When Scaling to 120+ Sites

Connection Pool Saturation

With HTTPX defaults, the 101st concurrent request queues until a connection frees. For 120 sites, this creates minor queuing; for 300 (Holehe's full module set), it becomes significant.

Symptoms: Progress bar moves in batches rather than steadily.

Mitigation: Filter modules to only necessary sites using --no-password-recovery (-NP) or a custom module list.

Timeout Configuration

The --timeout parameter (core.py:L94-L96) bounds each request:

# Aggressive timeout for fast networks

holehe user@example.com -T 5

# Generous timeout for VPN or high-latency paths

holehe user@example.com -T 20

A single hanging site with a 10-second default timeout delays result collection for that slot. With 120 sites, multiple slow endpoints compound this effect.

Rate Limiting and 429 Responses

Modules return rateLimit=True when detecting HTTP 429 or equivalent (core.py:L22-L26). Holehe continues other tasks, but the underlying TCP connection was already consumed.

Detection: Use --no-clear to preserve the progress bar and watch for stalls.

Response: Consider a custom wrapper with controlled delays:

import trio

async def throttled_launch(function, email, client, out):
    await trio.sleep(0.1)  # 100ms stagger between starts

    await launch_module(function, email, client, out)

CPU and Memory Characteristics

Holehe's workload is I/O-bound, not CPU-bound. Typical CPU usage stays below 5% on modern hardware because the event loop spends nearly all time awaiting network responses.

Implications:

• Adding CPU-intensive parsing (e.g., complex HTML extraction) disrupts this balance
• Memory usage scales with the number of concurrent responses held in memory
• The TrioProgress instrument (instruments.py:L8-L10) adds negligible overhead via tqdm updates

Practical Optimization Strategies

1. Reduce Module Count

Holehe discovers ~300 modules by default via import_submodules("holehe.modules") (core.py:L36-L47). For 120-site targets, explicitly select relevant modules:

from holehe.core import import_submodules, get_functions

modules = import_submodules("holehe.modules")

# Filter to specific sites

target_sites = ['github', 'twitter', 'instagram', 'linkedin']
filtered = {k: v for k, v in modules.items() if any(t in k for t in target_sites)}

2. Monitor with Instruments

The TrioProgress class provides real-time visibility. A flat progress bar indicates mass rate-limiting or network congestion.

holehe user@example.com --no-clear

3. Programmatic Control

For production scanning, replicate Holehe's core logic with custom constraints:

import httpx
import trio
from holehe.core import import_submodules, launch_module
from holehe.instruments import TrioProgress

async def controlled_scan(email, sites, max_concurrent=50, timeout=10):
    modules = import_submodules("holehe.modules")
    funcs = [
        modules[m].__dict__[m.split(".")[-1]]
        for m in modules
        if m.split(".")[-1] in sites
    ]
    
    client = httpx.AsyncClient(
        timeout=timeout,
        limits=httpx.Limits(max_connections=max_concurrent)
    )
    
    out = []
    instrument = TrioProgress(len(funcs))
    trio.lowlevel.add_instrument(instrument)
    
    # Use semaphore for controlled concurrency

    sem = trio.Semaphore(max_concurrent)
    
    async def bounded_launch(fn):
        async with sem:
            await launch_module(fn, email, client, out)
    
    async with trio.open_nursery() as n:
        for fn in funcs:
            n.start_soon(bounded_launch, fn)
    
    trio.lowlevel.remove_instrument(instrument)
    await client.aclose()
    return out

Performance Checklist for 120+ Sites

Factor	Default	Recommended for 120+ Sites
HTTPX connections	~100	150-200 if customizing
Timeout	10s	5-15s based on network
Modules loaded	~300	Filter to 50-150 relevant
Stagger/delay	None	50-100ms if rate-limited
Progress visibility	Clearing	Use --no-clear

Summary

• Holehe uses a shared HTTPX async client and Trio nurseries for concurrent site checking, with architecture defined in holehe/core.py
• Connection pool defaults (~100) may bottleneck 120+ simultaneous requests; customize httpx.Limits for higher throughput
• Every module starts as an immediate coroutine, so reducing the module count via filtering or -NP directly improves performance
• Timeout tuning (-T) balances completeness against speed—aggressive values suit fast networks, generous values suit VPNs
• Rate limiting manifests as progress bar stalls; use --no-clear to detect and consider custom delays for throttled targets
• CPU usage remains minimal; focus optimizations on I/O concurrency and network behavior

Frequently Asked Questions

How many concurrent connections does Holehe use by default?

Holehe relies on HTTPX's default connection limits, approximately 100 concurrent keepalive connections. For 120+ sites, this creates mild queuing. The shared client is instantiated at core.py:L13-L14 without explicit limits, so the final behavior depends on HTTPX's internal defaults for your version.

Can I limit how many sites run simultaneously?

Holehe does not expose a built-in concurrency limiter. All modules start immediately via nursery.start_soon() in core.py:L18-L21. To enforce limits, use a trio.Semaphore wrapper as shown in the programmatic example above, or reduce the module count through filtering.

Why does Holehe slow down after 100-150 sites?

The progress bar often stalls when rate limits trigger across multiple targets. Each 429 response consumes a connection slot and delays result collection. Check for this pattern with --no-clear and consider targeting fewer sites or adding inter-request delays.

Is Holehe suitable for scanning thousands of email addresses?

Holehe is optimized for breadth (many sites) over volume (many emails). For large email lists, wrap Holehe calls with your own orchestration that respects remote rate limits, or implement backoff strategies in custom modules to avoid IP bans.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}