# How megadose/holehe's Module System Dynamically Imports and Executes Website Checks

> Discover how megadose/holehe's module system dynamically imports and executes website checks using pkgutil.walk_packages and Trio for concurrent async operations.

- Repository: [Palenath/holehe](https://github.com/megadose/holehe)
- Tags: internals
- Published: 2026-08-15

---

**Holehe dynamically imports and executes website‑check modules using `pkgutil.walk_packages` to discover all Python files under `holehe.modules`, extracts matching async functions, and runs them concurrently via a shared `httpx.AsyncClient` managed by Trio nurseries.**

Holehe is an OSINT tool that checks whether an email address is registered across hundreds of websites. Its architecture relies entirely on a dynamic module system that eliminates hard‑coded site lists—new checks are added simply by dropping Python files into the `holehe/modules/` tree. This article examines exactly how this system works according to the megadose/holehe source code.

## Module Discovery: Walking the Package Tree

The dynamic import process begins in [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py) with the `import_submodules` function at lines 37–47. This function receives a package name string—typically `"holehe.modules"`—and converts it into a real module using `importlib.import_module`.

Once the base package is loaded, `pkgutil.walk_packages` traverses every sub‑package and module, such as `holehe.modules.social_media.twitter` or `holehe.modules.porn.pornhub`. For each discovered file, the function executes `importlib.import_module(full_name)` and stores the resulting module objects in a dictionary keyed by their dotted names.

```python

# holehe/core.py (simplified from L37-L47)

import pkgutil
import importlib

def import_submodules(package_name):
    package = importlib.import_module(package_name)
    results = {}
    for _, name, is_pkg in pkgutil.walk_packages(package.__path__, package.__name__ + '.'):
        full_name = package.__name__ + '.' + name
        results[full_name] = importlib.import_module(full_name)
    return results

```

## Function Extraction: Finding the Callable Checks

After discovery, `get_functions` at lines 50–63 filters and extracts the actual website‑check functions. It operates on three rules:

1. **Depth check**: Only modules with more than three dotted components are considered (e.g., `holehe.modules.social_media.twitter` qualifies; `holehe.modules` does not).
2. **Name matching**: The function name must match the final component of the module path—`twitter` for `holehe.modules.social_media.twitter`.
3. **Attribute lookup**: The matching attribute is retrieved via `getattr(module, name.split('.')[-1])`.

The collected callables are returned as the `websites` list—the complete set of checks to execute.

```python

# holehe/core.py (simplified from L50-L63)

def get_functions(modules):
    websites = []
    for path, module in modules.items():
        splited = path.split('.')
        if len(splited) > 3:
            try:
                websites.append(getattr(module, splited[-1]))
            except AttributeError:
                pass
    return websites

```

## Concurrency Setup: Shared HTTP Client

Before execution, `maincore` prepares a single `httpx.AsyncClient` at lines 12–14. This client is passed to every website‑check function, eliminating the overhead of connection pooling per site. The client supports HTTP/2 and connection reuse, critical for checking hundreds of services efficiently.

```python

# holehe/core.py (L12-L14)

import httpx

client = httpx.AsyncClient(http2=True, timeout=10)

```

## Async Execution: Trio Nurseries and launch_module

The execution phase occurs in `maincore` at lines 66–78. A **Trio nursery** schedules each website check concurrently via `launch_module`. This helper coroutine:

- Accepts the target `email`, shared `client`, and output list `out`
- Calls `await module(email, client, out)` where `module` is the site‑specific function
- Catches exceptions and appends a standardized error result

```python

# holehe/core.py (simplified from L66-L78)

import trio

async def launch_module(module, email, client, out):
    try:
        await module(email, client, out)
    except Exception as e:
        name = module.__name__
        out.append({
            "name": name,
            "domain": None,
            "rateLimit": False,
            "error": True,
            "exists": None,
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })

async def maincore(email):
    modules = import_submodules("holehe.modules")
    websites = get_functions(modules)
    out = []
    async with httpx.AsyncClient(http2=True, timeout=10) as client:
        async with trio.open_nursery() as nursery:
            for website in websites:
                nursery.start_soon(launch_module, website, email, client, out)
    # Results sorted and displayed (L23-L31)

    return sorted(out, key=lambda x: x["name"])

```

## Uniform Module Interface: The Contract Every Check Must Follow

Every module under `holehe/modules/` implements an **identical async signature**. This contract—`async def sitename(email, client, out)`—is what enables the dynamic system to treat all checks uniformly. A typical implementation:

```python

# holehe/modules/social_media/twitter.py (representative pattern)

async def twitter(email, client, out):
    try:
        # Site-specific HTTP logic

        response = await client.post(
            "https://api.twitter.com/i/users/email_available.json",
            data={"email": email}
        )
        data = response.json()
        
        out.append({
            "name": "twitter",
            "domain": "twitter.com",
            "rateLimit": "rate_limit" in str(data),
            "error": False,
            "exists": not data.get("valid", True),
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })
    except Exception:
        out.append({
            "name": "twitter",
            "domain": "twitter.com",
            "rateLimit": False,
            "error": True,
            "exists": None,
            "emailrecovery": None,
            "phoneNumber": None,
            "others": None,
        })

```

## Adding New Website Checks: Zero Core Changes Required

The dynamic architecture means **no modifications to [`core.py`](https://github.com/megadose/holehe/blob/main/core.py) are needed** when adding support for new sites. The process is:

1. Create a file at `holehe/modules/<category>/newsite.py`
2. Define `async def newsite(email, client, out)` matching the filename
3. Implement site-specific HTTP logic following the standardized output format
4. Restart Holehe—the module is automatically discovered and executed

## Summary

- **Discovery**: `import_submodules` in [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py#L37-L47) uses `pkgutil.walk_packages` to find all modules under `holehe.modules`
- **Extraction**: `get_functions` (L50-L63) filters by depth and extracts functions matching their filenames
- **Concurrency**: A shared `httpx.AsyncClient` and Trio nurseries (L66-78) execute hundreds of checks in parallel
- **Uniformity**: Every module follows the `async def sitename(email, client, out)` contract
- **Extensibility**: New checks are added by creating files—no registry or core code changes required

## Frequently Asked Questions

### How does Holehe find new modules automatically without a central registry?

Holehe uses `pkgutil.walk_packages` in `import_submodules` ([`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py#L37-L47) to traverse the `holehe.modules` package tree at runtime. Any Python file placed in this hierarchy is discovered and imported dynamically, eliminating the need for manual registration lists.

### Why does Holehe use Trio instead of asyncio for concurrency?

The megadose/holehe codebase specifically employs **Trio nurseries** (`trio.open_nursery`) at lines 66-78 of [`core.py`](https://github.com/megadose/holehe/blob/main/core.py) to spawn concurrent tasks. Trio provides structured concurrency with automatic error propagation and cancellation, which simplifies error handling across hundreds of parallel HTTP requests compared to raw asyncio.

### What happens if a website check raises an exception?

The `launch_module` helper at [`holehe/core.py`](https://github.com/megadose/holehe/blob/main/holehe/core.py#L66-L78) wraps each module call in a `try/except` block. Exceptions are caught and converted to a standardized error dictionary with `"error": True`, ensuring the output list remains consistent and the overall execution continues uninterrupted.

### Can I add a check that requires authentication or special headers?

Yes. Because each module receives the shared `httpx.AsyncClient`, you can configure custom headers, authentication, or request parameters within your module file. The client is fully featured and supports HTTP/2, timeouts, and all standard `httpx` functionality.