How to Manage Module-Specific Rate Limits and Errors in Large-Scale Holehe Scans

Use the frequent_rate_limit module flag, central launch_module wrapper, and standardized result dictionary to handle rate limits and errors at scale.

Holehe is an async OSINT tool that checks email addresses against hundreds of online services simultaneously. When running large-scale scans, you need robust mechanisms to handle rate limiting and module failures without crashing the entire run. The tool handles this through a three-layer architecture: per-module rate-limit flags, a centralized execution wrapper, and uniform result reporting.

Understanding the Rate-Limit Architecture

Every Holehe module follows a strict contract. It receives an email address, an HTTP client, and an output list, then appends a standardized result dictionary. This design lets the core runner scale to hundreds of services without custom logic for each site.

The Standard Result Dictionary

All modules return the same keys, enabling consistent downstream processing:

Key	Purpose
name	Human-readable service name (e.g., "twitter")
domain	Service domain (e.g., "twitter.com")
rateLimit	True when HTTP 429 or equivalent detected
error	True for unexpected exceptions (network, parsing, etc.)
exists	True if email found on the service
emailrecovery, phoneNumber, others	Optional extracted data

This uniformity lives in holehe/core.py, which orchestrates module execution and result formatting.

Layer 1: Per-Module Rate-Limit Flag with frequent_rate_limit

Each module declares a frequent_rate_limit boolean at the top of its file. This flag signals whether the target service enforces aggressive throttling.

# holehe/modules/social_media/twitter.py#L9

frequent_rate_limit = False

Services with stricter limits set this to True:

# holehe/modules/social_media/wattpad.py

frequent_rate_limit = True  # This service needs back-off between requests

The runner uses this flag to inject controlled delays with await trio.sleep(..) between calls, preventing hard blocks on sensitive endpoints.

Layer 2: Central Error Handling in launch_module

All modules execute through launch_module in holehe/core.py (lines 66-78). This wrapper guarantees that no unhandled exception crashes the scan.

# holehe/core.py#L66-L78

async def launch_module(module, email, client, out):
    try:
        await module(email, client, out)
    except Exception:
        name = str(module).split('<function ')[1].split(' ')[0]
        out.append({
            "name": name,
            "domain": data[name],
            "rateLimit": False,
            "error": True,
            "exists": False,
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })

Key behaviors:

• Any exception becomes a structured error=True entry
• rateLimit remains False because the failure wasn't a 429 response
• The scan continues with remaining modules

Layer 3: Visual Result Classification in print_result

After execution, print_result (lines 22-36) color-codes outcomes for immediate operator awareness:

# holehe/core.py#L22-L36

if results["rateLimit"] and args.onlyused == False:
    websiteprint = print_color("[x] " + results["domain"], "yellow", args)
elif "error" in results.keys() and results["error"] and args.onlyused == False:
    websiteprint = print_color("[!] " + results["domain"], "red", args)

Symbol	Color	Meaning
[+]	Green	Email found (exists=True)
[-]	Magenta	Email not found (exists=False)
[x]	Yellow	Rate-limited (rateLimit=True)
[!]	Red	Error occurred (error=True)

Running Large-Scale Scans: CLI and Programmatic Examples

Command-Line Usage

holehe john.doe@example.com

Typical output showing all four states:

[+] twitter.com      # email found

[-] github.com       # not registered

[x] instagram.com    # HTTP 429 - rate limited

[!] taringa.com      # network timeout / parsing error

Programmatic Rate-Limit Filtering

import asyncio
import httpx
import holehe.core as hc

async def run(email):
    async with httpx.AsyncClient(timeout=10) as client:
        out = []
        # Load all site modules dynamically

        mods = hc.import_submodules('holehe.modules')
        sites = hc.get_functions(mods)
        
        # Execute each module

        for mod in sites:
            await hc.launch_module(mod, email, client, out)

    # Post-process: isolate rate-limited services

    limited = [r for r in out if r['rateLimit']]
    errors = [r for r in out if r['error']]
    
    print("Rate-limited:", [r['domain'] for r in limited])
    print("Errors:", [r['domain'] for r in errors])
    return out

asyncio.run(run("john.doe@example.com"))

Creating a New Module with Custom Rate-Limit Handling

When adding a service, implement explicit 429 detection and set the appropriate flag:

# holehe/modules/social_media/newservice.py

frequent_rate_limit = True  # Service throttles aggressively

async def newservice(email, client, out):
    try:
        r = await client.get(
            f"https://api.newservice.com/users/{email}",
            headers={"User-Agent": "Holehe/1.0"}
        )
        
        # Explicit rate-limit detection

        if r.status_code == 429:
            out.append({
                "name": "newservice",
                "domain": "newservice.com",
                "rateLimit": True,
                "error": False,
                "exists": False,
                "emailrecovery": None,
                "phoneNumber": None,
                "others": None,
            })
            return
            
        # Normal response processing

        data = r.json()
        out.append({
            "name": "newservice",
            "domain": "newservice.com",
            "rateLimit": False,
            "error": False,
            "exists": data.get("found", False),
            "emailrecovery": data.get("recovery_email"),
            "phoneNumber": None,
            "others": None,
        })
        
    except Exception:
        # Re-raise to let launch_module handle as error=True

        raise

Key Source Files for Rate-Limit Management

File	Critical Function
holehe/core.py	launch_module (exception wrapping), print_result (visual output), module loader
holehe/modules/social_media/twitter.py	Reference module showing frequent_rate_limit=False and standard result structure
holehe/modules/social_media/wattpad.py	Example with frequent_rate_limit=True for throttle-sensitive service
holehe/instruments.py	Trio progress instruments for async feedback during rate-limit sleeps

Scaling Strategies for Production Use

When managing module-specific rate limits and errors across thousands of targets:

• Group modules by frequent_rate_limit status to apply tiered concurrency limits
• Capture [x] and [!] outputs for retry queues with exponential back-off
• Monitor per-domain error rates to detect when services change their throttling behavior
• Use the standardized dictionary to feed results into SIEM or alerting pipelines

The declarative frequent_rate_limit flag lets you tune behavior without modifying core logic—ideal for operational environments where service limits change frequently.

Summary

• frequent_rate_limit boolean in each module signals aggressive throttling to the runner
• launch_module in holehe/core.py catches all exceptions and converts them to structured error=True entries
• Standard result dictionaries with rateLimit and error keys enable uniform processing across hundreds of services
• print_result color-codes [x] (yellow/rate-limited) and [!] (red/error) for immediate visual triage
• The architecture scales because new modules only need to set a flag and return the standard keys—no core changes required

Frequently Asked Questions

How does Holehe detect rate limits versus other errors?

Holehe relies on module-level HTTP status code inspection. When a module receives HTTP 429 (or equivalent), it sets rateLimit=True in the result dictionary. The central launch_module wrapper only sets error=True for unhandled exceptions that bubble up, not for explicitly handled 429 responses. This distinction appears in print_result as [x] versus [!].

Can I customize the delay for frequent_rate_limit=True modules?

The current implementation in holehe/core.py uses trio.sleep with a fixed short delay. To customize, you would modify the runner loop where it checks module.frequent_rate_limit before execution. The modular structure keeps this logic centralized, so per-service tuning only requires changes in one location.

What happens if a module crashes mid-execution?

The launch_module wrapper's try/except block guarantees graceful degradation. The exception is caught, a result entry with error=True is appended, and execution proceeds to the next module. No thrown exception aborts the entire scan, as implemented in holehe/core.py lines 66-78.

How do I retry rate-limited services after a cooldown period?

The current Holehe architecture doesn't include automatic retry. For production implementations, filter the output list for rateLimit=True entries, then re-run those specific modules after a delay. The standardized dictionary format makes this straightforward to implement in a wrapper script.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}