How Holehe Modules Are Discovered and Loaded Dynamically: A Deep Dive into the Python Import System

Holehe uses Python's pkgutil.walk_packages and importlib to recursively discover, import, and execute service-checking modules at runtime without manual registration.

Holehe's modular architecture makes it trivial to add new service checks: drop a Python file into the correct directory and the framework handles the rest. This article explains exactly how Holehe discovers and loads modules dynamically, based on the source code in megadose/holehe.

The Three-Stage Discovery Pipeline

The dynamic loading system in holehe/core.py operates through three coordinated stages: recursive package traversal, callable extraction, and asynchronous execution.

Stage 1: Recursive Package Walking with import_submodules

The entry point is import_submodules at lines 37-47 of holehe/core.py. This function leverages pkgutil.walk_packages to traverse the entire holehe.modules package tree.

import importlib
import pkgutil

def import_submodules(package, recursive=True):
    """Recursively import every submodule in the given package."""
    if isinstance(package, str):
        package = importlib.import_module(package)
    
    modules = {}
    for loader, name, is_pkg in pkgutil.walk_packages(package.__path__):
        full_name = f"{package.__name__}.{name}"
        modules[full_name] = importlib.import_module(full_name)
        
        if is_pkg and recursive:
            modules.update(import_submodules(full_name))
    
    return modules

Key implementation details:

• String-to-module conversion: Accepts either a module object or a dotted package name
• Full dotted names: Constructs complete paths like holehe.modules.social_media.twitter
• Controlled recursion: The recursive flag enables deep traversal into category subdirectories
• Eager import: Calls importlib.import_module immediately, not lazily

Stage 2: Extracting Callable Functions with get_functions

Raw module objects aren't directly executable. The get_functions implementation at lines 50-63 filters and transforms the discovered modules into callable async functions.

def get_functions(modules, args=None):
    """Extract service-checking functions from imported modules."""
    functions = []
    
    for dotted_name in modules:
        # Filter: only leaf modules (depth > 3)

        # e.g., holehe.modules.social_media.twitter qualifies

        # but holehe.modules.social_media does not

        if len(dotted_name.split(".")) > 3:
            mod = modules[dotted_name]
            fn_name = dotted_name.split(".")[-1]  # "twitter"

            functions.append(mod.__dict__[fn_name])
    
    return functions

The filtering logic enforces naming conventions:

• Modules must be nested at least three levels deep (holehe.modules.<category>.<module>)
• The async function inside must match the final component of the dotted name
• This convention is what enables automatic execution without configuration files

Stage 3: Asynchronous Execution via Trio Nursery

The final stage occurs in maincore (lines 105-108 of holehe/core.py). The collected functions are passed to a Trio nursery for concurrent execution.

async def launch_module(module_func, email, client, out):
    """Execute a single service-checking module."""
    try:
        await module_func(email, client, out)
    except Exception:
        # Uniform error handling ensures consistent output format

        out.append({
            "name": module_func.__name__,
            "domain": "...",          # resolved from module metadata

            "rateLimit": False,
            "error": True,
            "exists": False,
            # ... additional result fields

        })

Each module receives three arguments:

Parameter	Purpose
email	The target email address to check
client	Pre-configured HTTP client (httpx)
out	List to append result dictionaries

Required Module Structure for Auto-Discovery

For a module to be discovered and executed, it must follow strict conventions. The holehe/modules/social_media/twitter.py file demonstrates the pattern:

# File: holehe/modules/social_media/twitter.py

async def twitter(email, client, out):
    """
    Check if email is registered on Twitter.
    Must be async and match the filename.
    """
    # Perform HTTP request via provided client

    # Append result dictionary to out list

    out.append({
        "name": "twitter",
        "domain": "twitter.com",
        "method": "register",
        "frequent_rate_limit": False,
        "rateLimit": False,
        "exists": False,
        "emailrecovery": None,
        "phoneNumber": None,
        "others": None
    })

Critical requirements:

1. File location: holehe/modules/<category>/<service_name>.py
2. Function name: Must exactly match the filename (without .py)
3. Function signature: async def <name>(email, client, out)
4. Side effects only: Results appended to out; no return value used
5. Async mandatory: Function must be awaitable

Package Structure Supporting Dynamic Loading

The discovery mechanism relies on specific package layout conventions:

holehe/
├── core.py              # Discovery and orchestration logic

├── modules/
│   ├── __init__.py      # Package marker (can be empty)

│   ├── social_media/    # Category subdirectory

│   │   ├── __init__.py
│   │   ├── twitter.py   # Service module (auto-discovered)

│   │   └── github.py
│   └── software/        # Another category

│       ├── __init__.py
│       └── docker.py
├── localuseragent.py    # HTTP client configuration

└── instruments.py       # Progress instrumentation for Trio

The __init__.py files are essential—they mark directories as Python packages, enabling pkgutil.walk_packages to traverse them.

Performance and Extensibility Characteristics

Startup overhead: Module discovery occurs once at import time, not per execution. The import_submodules call populates a module cache that get_functions consumes.

Concurrency model: All discovered modules execute concurrently through Trio's nursery pattern. No module blocks another regardless of network latency.

Zero-registration extension: Adding a service requires zero code changes to core.py. The filesystem is the sole source of truth.

Memory footprint: All modules are imported eagerly. For installations with many unused modules, this trades memory for simplicity.

Summary

• import_submodules in holehe/core.py uses pkgutil.walk_packages for recursive package traversal
• get_functions filters modules by depth and extracts filename-matching async functions
• launch_module executes each function through a Trio nursery with standardized error handling
• Module conventions: Files must live at holehe/modules/<category>/<name>.py with matching async function signatures
• No registration required: The filesystem structure alone determines what gets executed

Frequently Asked Questions

How does Holehe find new modules without configuration files?

Holehe treats the holehe/modules/ directory as the configuration. The import_submodules function walks this package tree at runtime using pkgutil.walk_packages, importing every Python file it encounters. No central registry or manifest file exists—discovery is purely filesystem-based.

Why must the async function name match the filename?

The get_functions extractor at line 58 of holehe/core.py derives the function name directly from the final component of the dotted module path. This convention eliminates ambiguity when multiple functions exist in a module and provides a predictable, zero-configuration execution target.

Can modules be loaded lazily instead of at startup?

The current implementation in holehe/core.py uses eager import via importlib.import_module during the discovery walk. While Python's import system supports lazy loading patterns, Holehe's design prioritizes fast failure detection and simple error handling over deferred loading.

What happens if a module raises an exception during execution?

The launch_module function wraps each module call in a try-except block. Exceptions are caught and transformed into uniform result dictionaries with "error": True, ensuring one failing service doesn't crash the entire check run and preserving consistent output format across all modules.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}