How to Troubleshoot User‑Scanner Performance Issues: A Complete Technical Guide

User‑scanner performance bottlenecks typically stem from excessive concurrent HTTP requests, oversized timeout values, enabled "loud" modules, or unreliable proxies, all of which can be diagnosed by inspecting orchestrator.py and helpers.py configurations.

User‑scanner is an asynchronous username and email enumeration engine that executes site‑specific validators in parallel across hundreds of platforms. When scans slow down, hang, or consume excessive memory, the root cause usually lies in the concurrency controls or network stack defined in the core source files. This guide explains how to troubleshoot user‑scanner performance issues by analyzing the kaifcodec/user-scanner codebase and applying targeted optimizations.

Understand the Async Architecture

The scanning engine centers on user_scanner/core/orchestrator.py, which manages the lifecycle of every validation task. At lines 27–33, the orchestrator initializes a shared thread pool and sets MAX_CONCURRENT_REQUESTS = 60 as the default semaphore limit. This cap controls how many simultaneous HTTP requests can fly at once.

The HTTP client pool lives in the same file at lines 31–41. Here, httpx.Client instances are cached and keyed by (http2, proxy, verify) tuples to eliminate the overhead of constructing new clients for every request. Line 64 wraps each validator in asyncio.wait_for, applying a default timeout equal to get_global_timeout() + 10 seconds.

Proxy rotation is handled by the ProxyManager class in user_scanner/core/helpers.py (lines 70–112), while the LOUD_MODULES list in the same file defines which sites are skipped unless explicitly allowed.

Identify Common Performance Bottlenecks

Performance degradation in user‑scanner usually falls into five categories:

• Semaphore Saturation – Setting concurrency too high relative to your bandwidth or the remote sites’ rate limits causes queue buildup and timeouts.
• Excessive Timeout Windows – A large global timeout forces every validator to wait extended periods before aborting slow responses.
• Loud Module Activation – Enabling --allow-loud includes heavy targets like LinkedIn and Instagram that employ aggressive anti‑bot measures, inflating request count and latency.
• Faulty Proxy Lists – Proxies that fail to connect add retry overhead and backoff delays via the ProxyManager rotation logic.
• Full Category Loading – Invoking run_user_full loads every category module at once, which can exhaust memory on modest hardware when load_categories is called.

Diagnostic Techniques

Inspect Runtime Configuration

Check the currently loaded concurrency and timeout values without running a scan:

user-scanner --show-config

Or query the helpers module programmatically:

import user_scanner.core.helpers as h
print(h.MAX_CONCURRENT_REQUESTS, h.get_global_timeout())

Enable Verbose Logging

Run with the -v flag to surface per‑request URLs and response timings in the console output. The Result.get_console_output method appends timing metadata when verbose is true, revealing which validators hang.

user-scanner -u testuser -v

Benchmark Individual Modules

Isolate slow validators by wrapping them in a timing script:

import asyncio
import time
from pathlib import Path
from user_scanner.core.helpers import load_modules, get_scan_func

async def bench(module_path: Path):
    module = load_modules(module_path)[0]
    func = get_scan_func(module)
    start = time.time()
    await func("some_test_username")
    print(f"{module.__name__}: {time.time() - start:.2f}s")

# Example: benchmark Instagram validator

asyncio.run(bench(Path("user_scanner/user_scan/social/instagram.py")))

Validate Proxies

Remove dead proxies from rotation before launching a full scan:

user-scanner --validate-proxies proxy.txt

This invokes the validate_proxies function in helpers.py (lines 97–99), filtering the list to only responsive endpoints.

Limit Category Scope

Run a targeted subset to isolate problematic groups:

user-scanner -c social -c music -u someuser

This reduces the total task count and helps identify whether specific categories introduce latency.

Optimize Configuration for Speed

Apply these practical tweaks to restore optimal throughput:

• Reduce Concurrency – Lower the semaphore limit to 30 to reduce network pressure: user-scanner --concurrency 30 or call set_concurrency(30) programmatically.
• Tighten Timeouts – Force early aborts on slow sites: user-scanner --timeout 5.
• Skip Loud Modules – Omit --allow-loud (the default) or set auto_loud_single_module_prompt=False in config.json to avoid heavy JavaScript‑heavy targets.
• Curate Proxies – Supply a short, validated proxy list via --proxy-file fast-proxies.txt to minimize rotation latency.
• Disable HTTP/2 – Some remote servers perform poorly with HTTP/2; forcing HTTP/1.1 via --http2 false can improve stability.

Implementation Examples

Fast Command‑Line Scan

Run a high‑speed investigation with conservative limits:

user-scanner -u alice \
    --concurrency 20 \
    --timeout 4 \
    --no-loud \
    --no-nsfw

Programmatic Configuration

Adjust settings before invoking the engine:

from user_scanner.core.helpers import set_concurrency, set_global_timeout, ScanConfig
from user_scanner.core.orchestrator import run_user_full

# Tighten resource limits

set_concurrency(25)
set_global_timeout(5.0)

cfg = ScanConfig(allow_loud=False, no_nsfw=True, verbose=False)
results = run_user_full("bob", cfg)
for r in results:
    print(r)

Proxy Validation Workflow

Verify and load only responsive proxies:

from user_scanner.core.helpers import validate_proxies, set_proxy_manager, ScanConfig
from user_scanner.core.orchestrator import run_user_full

good_proxies = validate_proxies([
    "http://1.2.3.4:8080",
    "socks5://5.6.7.8:1080"
])
set_proxy_manager(proxies=good_proxies)

cfg = ScanConfig(allow_loud=False, verbose=True)
run_user_full("charlie", cfg)

Summary

• The default MAX_CONCURRENT_REQUESTS = 60 in orchestrator.py often needs reduction to prevent network saturation.
• Per‑module timeouts default to global timeout plus 10 seconds; lowering the global value cuts overall runtime.
• Loud modules defined in helpers.py drastically increase request volume and should remain disabled for speed.
• Proxy reliability directly impacts performance; always validate lists with --validate-proxies before scanning.
• Category filtering via -c flags reduces memory pressure and isolates slow validators.

Frequently Asked Questions

Why does user‑scanner hang when scanning certain usernames?

Hangs usually indicate validators waiting on unresponsive sites. Check the global timeout setting via get_global_timeout() and enable verbose mode (-v) to identify which module stalls. Reduce the timeout with --timeout or exclude the problematic site by omitting its category.

How can I reduce memory usage during large scans?

Avoid loading all categories simultaneously. Instead of run_user_full, invoke specific categories using -c flags, or set auto_update_status=False in config.json to prevent background refresh threads from consuming RAM.

What is the optimal concurrency setting for home networks?

Start with 20–30 concurrent requests (--concurrency 30). The default of 60 often saturates residential bandwidth or triggers remote rate limits, causing cascading timeouts that actually slow the total execution time.

Does enabling HTTP/2 improve scanning speed?

Not necessarily. While httpx supports HTTP/2 via the client pool in orchestrator.py, many target sites malfunction or throttle HTTP/2 connections. If you observe irregular latency, force HTTP/1.1 by passing --http2 false to the request layer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}