How the kaifcodec/user-scanner Orchestrator Handles Username Scan Batch Running

The orchestrator processes username scans in batches using asyncio for concurrency, capping at 60 simultaneous requests, with a Rich progress bar and configurable filtering for loud or NSFW modules.

The kaifcodec/user-scanner repository provides an asynchronous framework for checking username availability across hundreds of sites. Central to this system is the orchestrator in user_scanner/core/orchestrator.py, which coordinates batch execution while respecting user-defined limits and providing real-time feedback. Understanding how username scan batch running works requires examining its concurrency model, worker architecture, and result aggregation pipeline.

Core Architecture: The _run_batch Method

The heart of the orchestrator is _run_batch, an asynchronous method that accepts:

• modules: A list of loaded site modules to execute
• username: The target username string
• config: A ScanConfig dataclass with CLI-derived flags

This method implements cooperative multitasking with controlled parallelism to prevent overwhelming target sites or hitting local resource limits.

Concurrency Control via Semaphore

Before any work begins, _run_batch initializes an asyncio.Semaphore with MAX_CONCURRENT_REQUESTS (default 60) to enforce an upper bound on simultaneous network operations:

# From user_scanner/core/orchestrator.py, lines 90-94

semaphore = asyncio.Semaphore(MAX_CONCURRENT_REQUESTS)

This semaphore passes to every worker task, which must acquire it before executing the site's validation function. Users can override the default via the --concurrency CLI flag.

Real-Time Progress Visualization

The orchestrator integrates the Rich library for terminal UI feedback:

# Conceptual structure from lines 94-100

with Progress() as progress:
    task = progress.add_task(f"Scanning {username}...", total=len(modules))
    # ... task creation and monitoring

The progress bar updates dynamically, showing the site currently being processed and overall completion percentage.

Worker Execution Model

For each module in the batch, the orchestrator spawns an asyncio.Task via create_task (lines 107-112). These tasks execute _async_worker, a structured coroutine handling individual site checks.

The _async_worker Lifecycle

Located at lines 63-73 in user_scanner/core/orchestrator.py, _async_worker performs four critical operations:

1. Metadata extraction – Retrieves site_name and the validation function via helpers from user_scanner/core/helpers.py

2. Loud module filtering – Checks is_loud(module) and skips execution unless config.allow_loud is True. This prevents "noisy" sites that might notify the target user from running unexpectedly

3. Execution strategy selection – Determines whether the validation function is a coroutine or synchronous:

   • Async path: await func(username) for native async validators
   • Sync path: await loop.run_in_executor(_shared_executor, func, username) for blocking code, using a shared ThreadPoolExecutor

4. Exception normalization – Catches asyncio.TimeoutError and unexpected exceptions, wrapping them in Result.error objects with consistent metadata

# Simplified representation of execution path selection

if asyncio.iscoroutinefunction(func):
    result = await func(username)
else:
    result = await loop.run_in_executor(_shared_executor, func, username)

Result Aggregation and Display

As tasks complete, _run_batch processes them in completion order using asyncio.as_completed (lines 115-127):

• Category headers – Prints a grouped header the first time a result from a new category appears (when show_all is enabled or the result is visible)
• Rendering – Invokes result.show(configs) to apply user formatting preferences (verbose mode, color output, etc.)
• Collection – Accumulates all Result objects into a final list returned to the caller

This streaming approach ensures users see findings immediately rather than waiting for the entire batch to finish.

High-Level Orchestration Entry Points

The orchestrator exposes three convenience functions wrapping _run_batch:

Function	Purpose	Location
run_user_module	Scan single module or explicit list synchronously via asyncio.run	Lines 31-35
run_user_category	Load all modules in a category directory and execute batch	Lines 38-45
run_user_full	Pre-load all categories with global semaphore, stream results category-by-category	Lines 59-64

Single Module or List Execution

For targeted scans, run_user_module handles the event loop management:

from user_scanner.core.orchestrator import run_user_module
from user_scanner.core.helpers import ScanConfig

config = ScanConfig(allow_loud=False, show_all=True)
results = run_user_module("target_user", ["linkedin", "github"], config)

Category-Scoped Scanning

run_user_category leverages load_modules from helpers to dynamically discover validators:

from user_scanner.core.orchestrator import run_user_category
from user_scanner.core.helpers import ScanConfig
from pathlib import Path

social_path = Path(__file__).parent / "user_scanner" / "user_scan" / "social"
config = ScanConfig(no_nsfw=True)
results = run_user_category(social_path, "developer1", config)

Full Catalog Scanning

run_user_full implements the most complex orchestration, maintaining a single semaphore across all categories to honor global concurrency limits while providing category-grouped output.

Configuration and Filtering

The ScanConfig dataclass (defined in user_scanner/core/helpers.py) controls batch behavior:

Field	Effect on Batch Execution
allow_loud	Permits modules flagged as notification-triggering
no_nsfw	Excludes adult-content sites from loading
show_all	Displays negative findings and skipped results
verbose	Enables detailed error reporting and timing
concurrency	Overrides MAX_CONCURRENT_REQUESTS default

These flags pass through to _run_batch and _async_worker, affecting both module selection and result presentation.

Summary

• The orchestrator's _run_batch method in user_scanner/core/orchestrator.py implements username scan batch running with asyncio concurrency capped at 60 simultaneous requests
• Semaphore acquisition ensures respectful scanning that won't overwhelm target infrastructure
• _async_worker handles per-site execution with automatic sync/async dispatch and exception normalization
• Rich progress bars provide real-time visibility into batch completion status
• High-level helpers (run_user_module, run_user_category, run_user_full) offer flexible entry points for different scanning scopes
• ScanConfig flags enable fine-grained control over loud modules, NSFW content, and output verbosity

Frequently Asked Questions

What limits how many sites the orchestrator scans simultaneously?

The asyncio.Semaphore initialized with MAX_CONCURRENT_REQUESTS (default 60) controls concurrency. Users can override this via the --concurrency CLI flag, which passes through ScanConfig to the semaphore constructor.

How does the orchestrator handle synchronous site validators?

The _async_worker function detects synchronous validators using asyncio.iscoroutinefunction. Non-async functions execute in a shared ThreadPoolExecutor via loop.run_in_executor(), preventing blocking the event loop while maintaining uniform async semantics.

What happens when a site check times out or crashes?

_async_worker catches asyncio.TimeoutError and general exceptions, wrapping them in Result.error objects with site metadata preserved. These error results flow through the normal aggregation pipeline and render according to config.verbose settings.

Can I prevent "noisy" sites from running in a batch scan?

Yes. Set allow_loud=False in ScanConfig (or omit --allow-loud on CLI). The orchestrator checks is_loud(module) before executing each validator and skips loud modules unless explicitly permitted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}