How megadose/holehe's Module System Dynamically Imports and Executes Website Checks

Holehe dynamically imports and executes website‑check modules using pkgutil.walk_packages to discover all Python files under holehe.modules, extracts matching async functions, and runs them concurrently via a shared httpx.AsyncClient managed by Trio nurseries.

Holehe is an OSINT tool that checks whether an email address is registered across hundreds of websites. Its architecture relies entirely on a dynamic module system that eliminates hard‑coded site lists—new checks are added simply by dropping Python files into the holehe/modules/ tree. This article examines exactly how this system works according to the megadose/holehe source code.

Module Discovery: Walking the Package Tree

The dynamic import process begins in holehe/core.py with the import_submodules function at lines 37–47. This function receives a package name string—typically "holehe.modules"—and converts it into a real module using importlib.import_module.

Once the base package is loaded, pkgutil.walk_packages traverses every sub‑package and module, such as holehe.modules.social_media.twitter or holehe.modules.porn.pornhub. For each discovered file, the function executes importlib.import_module(full_name) and stores the resulting module objects in a dictionary keyed by their dotted names.

# holehe/core.py (simplified from L37-L47)

import pkgutil
import importlib

def import_submodules(package_name):
    package = importlib.import_module(package_name)
    results = {}
    for _, name, is_pkg in pkgutil.walk_packages(package.__path__, package.__name__ + '.'):
        full_name = package.__name__ + '.' + name
        results[full_name] = importlib.import_module(full_name)
    return results

Function Extraction: Finding the Callable Checks

After discovery, get_functions at lines 50–63 filters and extracts the actual website‑check functions. It operates on three rules:

1. Depth check: Only modules with more than three dotted components are considered (e.g., holehe.modules.social_media.twitter qualifies; holehe.modules does not).
2. Name matching: The function name must match the final component of the module path—twitter for holehe.modules.social_media.twitter.
3. Attribute lookup: The matching attribute is retrieved via getattr(module, name.split('.')[-1]).

The collected callables are returned as the websites list—the complete set of checks to execute.

# holehe/core.py (simplified from L50-L63)

def get_functions(modules):
    websites = []
    for path, module in modules.items():
        splited = path.split('.')
        if len(splited) > 3:
            try:
                websites.append(getattr(module, splited[-1]))
            except AttributeError:
                pass
    return websites

Concurrency Setup: Shared HTTP Client

Before execution, maincore prepares a single httpx.AsyncClient at lines 12–14. This client is passed to every website‑check function, eliminating the overhead of connection pooling per site. The client supports HTTP/2 and connection reuse, critical for checking hundreds of services efficiently.

# holehe/core.py (L12-L14)

import httpx

client = httpx.AsyncClient(http2=True, timeout=10)

Async Execution: Trio Nurseries and launch_module

The execution phase occurs in maincore at lines 66–78. A Trio nursery schedules each website check concurrently via launch_module. This helper coroutine:

• Accepts the target email, shared client, and output list out
• Calls await module(email, client, out) where module is the site‑specific function
• Catches exceptions and appends a standardized error result

# holehe/core.py (simplified from L66-L78)

import trio

async def launch_module(module, email, client, out):
    try:
        await module(email, client, out)
    except Exception as e:
        name = module.__name__
        out.append({
            "name": name,
            "domain": None,
            "rateLimit": False,
            "error": True,
            "exists": None,
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })

async def maincore(email):
    modules = import_submodules("holehe.modules")
    websites = get_functions(modules)
    out = []
    async with httpx.AsyncClient(http2=True, timeout=10) as client:
        async with trio.open_nursery() as nursery:
            for website in websites:
                nursery.start_soon(launch_module, website, email, client, out)
    # Results sorted and displayed (L23-L31)

    return sorted(out, key=lambda x: x["name"])

Uniform Module Interface: The Contract Every Check Must Follow

Every module under holehe/modules/ implements an identical async signature. This contract—async def sitename(email, client, out)—is what enables the dynamic system to treat all checks uniformly. A typical implementation:

# holehe/modules/social_media/twitter.py (representative pattern)

async def twitter(email, client, out):
    try:
        # Site-specific HTTP logic

        response = await client.post(
            "https://api.twitter.com/i/users/email_available.json",
            data={"email": email}
        )
        data = response.json()
        
        out.append({
            "name": "twitter",
            "domain": "twitter.com",
            "rateLimit": "rate_limit" in str(data),
            "error": False,
            "exists": not data.get("valid", True),
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })
    except Exception:
        out.append({
            "name": "twitter",
            "domain": "twitter.com",
            "rateLimit": False,
            "error": True,
            "exists": None,
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })

Adding New Website Checks: Zero Core Changes Required

The dynamic architecture means no modifications to core.py are needed when adding support for new sites. The process is:

1. Create a file at holehe/modules/<category>/newsite.py
2. Define async def newsite(email, client, out) matching the filename
3. Implement site-specific HTTP logic following the standardized output format
4. Restart Holehe—the module is automatically discovered and executed

Summary

• Discovery: import_submodules in holehe/core.py uses pkgutil.walk_packages to find all modules under holehe.modules
• Extraction: get_functions (L50-L63) filters by depth and extracts functions matching their filenames
• Concurrency: A shared httpx.AsyncClient and Trio nurseries (L66-78) execute hundreds of checks in parallel
• Uniformity: Every module follows the async def sitename(email, client, out) contract
• Extensibility: New checks are added by creating files—no registry or core code changes required

Frequently Asked Questions

How does Holehe find new modules automatically without a central registry?

Holehe uses pkgutil.walk_packages in import_submodules (holehe/core.py to traverse the holehe.modules package tree at runtime. Any Python file placed in this hierarchy is discovered and imported dynamically, eliminating the need for manual registration lists.

Why does Holehe use Trio instead of asyncio for concurrency?

The megadose/holehe codebase specifically employs Trio nurseries (trio.open_nursery) at lines 66-78 of core.py to spawn concurrent tasks. Trio provides structured concurrency with automatic error propagation and cancellation, which simplifies error handling across hundreds of parallel HTTP requests compared to raw asyncio.

What happens if a website check raises an exception?

The launch_module helper at holehe/core.py wraps each module call in a try/except block. Exceptions are caught and converted to a standardized error dictionary with "error": True, ensuring the output list remains consistent and the overall execution continues uninterrupted.

Can I add a check that requires authentication or special headers?

Yes. Because each module receives the shared httpx.AsyncClient, you can configure custom headers, authentication, or request parameters within your module file. The client is fully featured and supports HTTP/2, timeouts, and all standard httpx functionality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}