How Holehe Handles Rate Limiting When Checking Websites

Holehe manages rate limiting through a dual-layer system of static module flags and dynamic response detection, implemented primarily in holehe/core.py, which throttles requests for sensitive services and gracefully skips platforms that return HTTP 429 status codes.

Holehe is an open-source OSINT (Open Source Intelligence) tool designed to check email address usage across hundreds of online services simultaneously. When performing bulk queries against diverse platforms with varying tolerance levels, the tool must implement robust rate limiting to avoid triggering anti-bot protections. According to the megadose/holehe source code, the project handles this challenge through a decentralized flag system combined with centralized dispatch logic that monitors real-time responses.

Module-Level Rate Limiting Flags

Each service module in Holehe contains metadata that signals its rate limiting behavior to the central engine. This design allows individual modules to self-declare their sensitivity levels before any network requests occur.

The frequent_rate_limit Boolean

Every module defines a frequent_rate_limit boolean field within its descriptor dictionary. When set to True, this flag alerts the core engine that the service enforces strict or unpredictable request limits. For example, in holehe/modules/transport/blablacar.py, this flag is enabled because Blablacar aggressively blocks rapid sequential requests. Similarly, holehe/modules/social_media/wattpad.py and holehe/modules/social_media/patreon.py declare this flag to indicate their strict throttling policies.

# Example module descriptor structure (simplified from blablacar.py)

module_descriptor = {
    "name": "blablacar",
    "frequent_rate_limit": True,  # Signals strict rate limiting

    "url": "https://www.blablacar.fr/",
    # ... other metadata

}

Dynamic rateLimit Tracking

Beyond static flags, modules also maintain a rateLimit attribute that the central engine updates dynamically. As seen in holehe/modules/social_media/twitter.py, this boolean field indicates whether the service has recently returned a rate-limit response. The engine checks this attribute before subsequent requests to determine whether to skip or delay further queries against that specific service.

Central Engine Implementation

The orchestration logic resides in holehe/core.py, which iterates over all enabled modules and applies rate limiting policies based on the flags and runtime state.

Request Dispatching and Throttling

Before executing any HTTP request, the central dispatcher examines the module's frequent_rate_limit flag. If the flag evaluates to True, the engine injects a deliberate delay—typically using time.sleep(1) or similar—to reduce the request velocity. This proactive throttling occurs before network contact, preventing the initial request that might trigger a block.

# Simplified logic from holehe/core.py

for module in loaded_modules:
    descriptor = module.get_descriptor()
    
    # Check for known aggressive rate limiters

    if descriptor.get("frequent_rate_limit", False):
        time.sleep(1)  # Throttle before request

    
    response = requests.get(
        descriptor["url"], 
        headers=headers, 
        timeout=10
    )

HTTP 429 Detection and Back-off

After receiving a response, the engine inspects the HTTP status code. When a service returns 429 (Too Many Requests) or similar rate-limit indicators, the engine immediately sets the module's rateLimit attribute to True. Rather than aborting the entire scan, Holehe records this state and continues processing remaining services. Subsequent iterations check this flag and either skip the service entirely or apply exponential back-off delays.

# Response handling logic within the request loop

if response.status_code == 429:
    descriptor["rateLimit"] = True
    # Log the rate limit and continue with next module

    continue

if response.ok:
    process_success(response)

Graceful Degradation Under Pressure

Holehe's architecture ensures that rate limiting on one platform never cascades into complete scan failure. When a service signals enforcement through HTTP 429 or when the rateLimit attribute is set, the tool simply marks that specific check as inconclusive and proceeds with the remaining modules. The final output aggregates all successful queries while transparently indicating which services were skipped due to rate limiting, maximizing intelligence gathering while respecting third-party API constraints.

Summary

• Static Flags: Modules declare frequent_rate_limit=True in descriptors (e.g., blablacar.py, wattpad.py) to signal known strict limiters.
• Dynamic Tracking: The rateLimit attribute in modules like twitter.py allows runtime detection of 429 responses.
• Central Throttling: holehe/core.py enforces delays before requests when flags are detected and manages the request loop.
• Graceful Handling: HTTP 429 responses trigger service skipping rather than scan termination, preserving partial results.

Frequently Asked Questions

What causes Holehe to delay requests to specific services?

Holehe delays requests when a module's descriptor contains frequent_rate_limit=True, indicating the service aggressively blocks rapid queries. The central engine in core.py checks this flag before each request and inserts a sleep period to reduce velocity.

Does Holehe stop scanning if one website returns HTTP 429?

No, Holehe implements graceful degradation. When a service returns HTTP 429, the engine sets the module's rateLimit attribute to True, skips that specific check, and continues processing remaining services without aborting the full scan.

Which Holehe modules are known for strict rate limiting?

Based on the source code, modules such as holehe/modules/transport/blablacar.py, holehe/modules/social_media/wattpad.py, and holehe/modules/social_media/patreon.py explicitly set frequent_rate_limit=True due to their strict request policies.

How does Holehe detect rate limiting in real-time?

The engine monitors HTTP response status codes within the request loop in core.py. Upon detecting status 429 or similar rate-limit indicators, it immediately updates the module's rateLimit boolean and excludes that service from subsequent immediate retry attempts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}