How Holehe Handles Rate Limiting Across Different Modules: Core Architecture Explained

Holehe manages rate limiting through a two-tier system where individual modules declare a frequent_rate_limit boolean flag while the central orchestrator in holehe/core.py enforces adaptive delays—approximately 0.5 seconds for flagged high-frequency modules and 2 seconds for standard ones—with additional backoff when modules return "rateLimit": True in their result dictionaries.

Holehe, the open-source OSINT tool maintained by megadose for checking email address existence across hundreds of platforms, implements a sophisticated rate limiting architecture to avoid service bans while maximizing throughput. The design separates rate limit awareness into individual modules while centralizing timing logic in the core execution engine. This approach allows holehe to handle rate limiting across different modules with precision, adapting sleep delays based on each service's specific constraints and runtime feedback.

Module-Level Rate Limit Declarations

Each holehe module explicitly declares its rate limiting behavior through two mechanisms: a static module-level frequent_rate_limit boolean constant and a dynamic "rateLimit" key in the returned dictionary. This dual approach allows the core orchestrator to adjust pacing strategy before making requests and react dynamically when services throttle individual calls.

High-Frequency Modules (frequent_rate_limit=True)

Services with aggressive API limits or strict bot detection flag themselves as high-frequency targets. In holehe/modules/social_media/wattpad.py (lines 9-24), the module sets frequent_rate_limit = True, signaling to the core that it requires specific pacing considerations. Similarly, the transport module in holehe/modules/transport/blablacar.py (lines 9-34) adopts this flag for its high-frequency endpoints, ensuring the orchestrator applies appropriate throttling to prevent IP bans.

Standard Modules with Runtime Detection

Some modules maintain frequent_rate_limit = False but implement dynamic detection for specific endpoints that trigger throttling. The Xing module in holehe/modules/social_media/xing.py (lines 9-23) follows this pattern, returning "rateLimit": True in its result dictionary only when the service explicitly signals throttling. This allows the core to apply ad-hoc backoff for specific failed calls without slowing down the entire module's execution.

Core Orchestration and Adaptive Delays

The central execution logic resides in holehe/core.py, where the check_email function iterates through selected modules and implements the pacing strategy. Rather than hardcoding delays into each module, the core reads the frequent_rate_limit flag and applies standardized sleep intervals using asyncio.sleep().

When processing a module with frequent_rate_limit=True, the core inserts a short pause of approximately 0.5 seconds before the subsequent request. For modules with the flag set to False, it enforces a longer 2-second delay to stay under typical service limits. If a module returns "rateLimit": True in its output dictionary, the core triggers an additional backoff mechanism, often doubling the wait time for that specific call to accommodate the service's temporary restrictions.

Additionally, holehe/localuseragent.py supplies realistic User-Agent headers for all HTTP requests, reducing the baseline chance of services triggering rate limits in the first place.

Implementation Examples

The following examples demonstrate how holehe's rate limiting architecture operates in practice:

# holehe/modules/social_media/wattpad.py

# Lines 9-24 declare the module-level flag

frequent_rate_limit = True

async def wattpad(email, client, out):
    # ... request logic ...

    if response.status_code == 429:
        out.append({"name": "wattpad", "rateLimit": True, "exists": False})
    else:
        out.append({"name": "wattpad", "rateLimit": False, "exists": True})

# CLI usage automatically respects all rate limiting flags

# The CLI creates an aiohttp ClientSession and iterates over 

# enabled modules with appropriate delays

$ holehe -e user@example.com

# Programmatic usage with adaptive throttling

import asyncio
from holehe.core import check_email

async def main():
    results = await check_email(
        email="user@example.com",
        modules=["wattpad", "xing", "blablacar"],
    )
    for mod, data in results.items():
        print(f"{mod}: rate limited? {data.get('rateLimit')}")

asyncio.run(main())

In the programmatic example, check_email automatically respects each module's frequent_rate_limit flag and inserts the appropriate await asyncio.sleep() delays between asynchronous calls. The function aggregates results while transparently handling the pacing logic, allowing developers to query multiple services without manual throttling code.

Summary

• Module declarations: Each holehe module declares frequent_rate_limit as True or False to indicate its typical API strictness, with services like Wattpad and Blablacar flagging themselves as high-frequency targets that need specific handling.
• Centralized pacing: The holehe/core.py orchestrator reads these flags and enforces standardized delays—approximately 0.5 seconds for modules with frequent_rate_limit=True and 2 seconds for standard modules.
• Dynamic backoff: When individual calls return "rateLimit": True, the core applies additional exponential backoff, doubling wait times for specific failed requests without affecting the global module pacing.
• Async architecture: All delays use asyncio.sleep() to maintain non-blocking execution across hundreds of potential service checks, maximizing throughput while respecting service constraints.

Frequently Asked Questions

What is the difference between frequent_rate_limit and the rateLimit return value?

The frequent_rate_limit boolean is a static module-level constant defined in files like holehe/modules/social_media/wattpad.py that tells the core orchestrator to apply specific throttling behavior for the entire module. The rateLimit key in the return dictionary is a dynamic runtime flag indicating that a specific request encountered throttling. The former guides proactive pacing strategy, while the latter triggers reactive backoff for individual failed calls.

Where does holehe implement the sleep delays between requests?

Holehe implements all sleep delays in holehe/core.py rather than in individual modules. The core reads each module's frequent_rate_limit flag and applies asyncio.sleep() with durations of approximately 0.5 seconds for high-frequency modules and 2 seconds for standard modules, plus additional backoff when rateLimit=True appears in results.

How can I check if a specific module triggered rate limiting in my results?

When calling check_email programmatically, examine the "rateLimit" key in each module's result dictionary. If the value is True, the service temporarily blocked that request, and holehe automatically applied backoff before returning the result to your code.

Can I override the default rate limiting behavior for specific modules?

Currently, holehe does not expose public API parameters to override the frequent_rate_limit flags defined in module files. The delays are hardcoded in core.py based on the module's declared flag, ensuring consistent behavior across the tool's OSINT investigations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}