How Holehe Dynamically Imports and Loads All Website Modules at Runtime: A Deep Dive into Its Plugin Architecture

Holehe discovers and loads every site-specific module automatically using a utility called import_submodules located in holehe/core.py, enabling zero-configuration plugin discovery.

The OSINT tool holehe by megadose supports hundreds of websites without requiring manual registration of each new module. This dynamic module loading system lets contributors add new site checks simply by creating Python files in the appropriate subdirectory. Understanding this mechanism reveals how to extend holehe or implement similar auto-discovery patterns in your own Python projects.

The import_submodules Utility: Core of Dynamic Loading

All module discovery happens through a single function defined at lines 37‑47 in holehe/core.py. The import_submodules function implements a recursive package walker that transforms the static holehe/modules directory tree into a dictionary of loaded module objects.

How the Discovery Works

The utility combines two standard library modules: pkgutil for package introspection and importlib for runtime importing:

import importlib, pkgutil

def import_submodules(package, recursive=True):
    """Import every submodule of *package* and return a dict."""
    if isinstance(package, str):
        package = importlib.import_module(package)
    results = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = f"{package.__name__}.{name}"
        results[full_name] = importlib.import_module(full_name)
        if recursive and is_pkg:
            results.update(import_submodules(full_name))
    return results

pkgutil.walk_packages yields three-tuples for every entry in the package: the loader object, the module name, and a boolean is_pkg flag. This iteration covers both .py files and subdirectories containing __init__.py.

Building the Full Module Name

For each discovered entry, the function constructs the fully qualified dotted name by concatenating package.__name__ + '.' + name. This preserves the nested structure so that twitter.py inside holehe/modules/social_media/ becomes holehe.modules.social_media.twitter.

Recursive Loading of Nested Packages

The recursive=True default ensures holehe handles arbitrary directory depth. When is_pkg is True, import_submodules calls itself on that sub-package and merges results with results.update().

This design means the directory structure directly determines the module hierarchy:

holehe/modules/
├── social_media/
│   ├── __init__.py
│   ├── twitter.py          → holehe.modules.social_media.twitter
│   └── instagram.py        → holehe.modules.social_media.instagram
├── shopping/
│   ├── __init__.py
│   ├── amazon.py           → holehe.modules.shopping.amazon
│   └── ebay.py             → holehe.modules.shopping.ebay
└── __init__.py

No configuration file maps these paths to module names—the filesystem is the source of truth.

Runtime Invocation in maincore

The dynamic import system activates during the main execution flow. At lines 206‑208 in holehe/core.py, the maincore function orchestrates the loading:

modules = import_submodules("holehe.modules")
websites = get_functions(modules, args)

The two-phase process:

1. import_submodules("holehe.modules") returns a dictionary where keys are full module names and values are actual module objects. Every .py file under holehe/modules is now imported and available.

2. get_functions(modules, args) filters and transforms these modules into callable coroutines. It scans each module for functions matching the expected signature and selects those relevant to the command-line arguments.

From Module to Callable: Extracting Site Checkers

Each site module exports an async function with the same name as the module file. For example, holehe/modules/social_media/twitter.py contains:

async def twitter(email, client, out):
    # HTTP request to Twitter's password recovery endpoint

    # ...

    out.append({
        "name": "twitter",
        "domain": "twitter.com",
        "exists": True,
        "emailrecovery": "t****@gmail.com",
        # ...

    })

The get_functions utility inspects the imported modules using introspection, identifies these async functions, and returns them as a list of callables. These are then launched concurrently with Trio for efficient asynchronous execution against the target email address.

Performance and Scalability Characteristics

The import_submodules approach carries specific runtime implications:

• Import-time overhead occurs once at startup. All modules load regardless of whether they'll be used for a particular scan.
• Memory footprint includes every site module's code and dependencies in the Python process.
• Zero runtime discovery cost during actual OSINT checks—modules are already loaded and referenced.

For a tool like holehe with hundreds of modules, this tradeoff favors simplicity and contributor ergonomics over minimal memory usage.

Extending Holehe: Adding New Site Modules

The dynamic import architecture enables frictionless contribution. To add support for a new website:

1. Create a Python file under an appropriate category in holehe/modules/ (or create a new category directory with __init__.py).

2. Implement an async function matching the expected signature:

# holehe/modules/social_media/newsite.py

import httpx

async def newsite(email, client, out):
    """Check if email is registered on newsite.com."""
    try:
        r = await client.get(
            f"https://newsite.com/api/check_email?email={email}"
        )
        exists = r.status_code == 200
    except Exception:
        exists = False
    
    out.append({
        "name": "newsite",
        "domain": "newsite.com",
        "exists": exists,
        "emailrecovery": None,
        "phoneNumber": None,
        "others": None,
    })

3. No registration step required—the next run of holehe automatically includes newsite in its module dictionary.

Summary

• import_submodules in holehe/core.py (lines 37‑47) implements recursive dynamic importing using pkgutil.walk_packages and importlib.import_module.
• Package tree walking generates full dotted names for every Python file under holehe/modules, enabling unlimited nesting.
• Runtime loading happens in maincore (lines 206‑208), producing a dictionary of all site modules without manual registration.
• get_functions extracts async coroutines from loaded modules, preparing them for concurrent Trio execution.
• Contributor workflow requires only adding a Python file—no configuration changes or registry updates.

Frequently Asked Questions

How does holehe find modules without a central registry?

Holehe treats the filesystem as the registry. The pkgutil.walk_packages function enumerates all subdirectories and Python files under holehe/modules, and importlib.import_module loads each one by its computed dotted name. This eliminates the need for manual list maintenance.

Can holehe load modules from outside its package directory?

Not with the default import_submodules implementation. The function specifically walks package.__path__, which is constrained to the holehe/modules directory tree. Extending to external plugin directories would require modifying the path or implementing a separate plugin discovery mechanism.

What happens if a module has syntax errors or import failures?

importlib.import_module will raise an ImportError or SyntaxError during import_submodules execution, causing holehe to crash at startup. The current implementation does not include error isolation for individual module failures.

Why use recursive loading instead of flat module imports?

Recursion preserves the categorical organization of site modules (social_media, shopping, etc.) while keeping the loading code simple. It also ensures that future category additions require zero changes to the import machinery—just create a new subdirectory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}