# How HTTPX and curl_cffi Sessions Are Cached in the User-Scanner Repository

> Discover how httpx and curl_cffi sessions are cached globally in the user-scanner repository using thread-safe singleton patterns for efficient session management.

- Repository: [Kaif/user-scanner](https://github.com/kaifcodec/user-scanner)
- Tags: internals
- Published: 2026-09-02

---

**Both HTTPX and curl_cffi sessions are cached globally using thread-safe singleton patterns—HTTPX via a lazily-initialized `httpx.Client` in [`helpers.py`](https://github.com/kaifcodec/user-scanner/blob/main/helpers.py), and curl_cffi via a dictionary keyed by `(impersonate, proxy)` tuples in [`impersonate.py`](https://github.com/kaifcodec/user-scanner/blob/main/impersonate.py).**

The User-Scanner project performs high-volume username and email validation across dozens of platforms. To avoid the latency and resource exhaustion of establishing new TCP connections for every request, the codebase implements **process-wide session caching** for both its standard HTTP client (HTTPX) and its TLS-impersonating client (curl_cffi). This design is critical when running with the CLI `--concurrency` flag, where multiple threads execute validations simultaneously.

---

## HTTPX Session Caching: The Global Client Pattern

In [`user_scanner/core/helpers.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scanner/core/helpers.py), the repository maintains a single `httpx.Client` instance that lives for the entire process lifetime. This provides **connection pool reuse** transparently to all synchronous validators.

### Implementation Details

The global client is stored in `_httpx_client` and protected by `_client_lock` to ensure thread-safe initialization:

```python
import threading
import httpx

_httpx_client: httpx.Client | None = None
_client_lock = threading.Lock()


def get_httpx_client() -> httpx.Client:
    """Return a lazily-initialised global `httpx.Client`."""
    global _httpx_client
    if _httpx_client is None:
        with _client_lock:
            if _httpx_client is None:
                timeout = get_timeout()
                proxy = get_proxy()
                _httpx_client = httpx.Client(timeout=timeout, proxies=proxy)
    return _httpx_client

```

The **double-checked locking pattern** ensures that only one thread creates the client even when multiple threads race during startup.

### Using the Cached Client

Validators call `generic_validate`, which retrieves the global client automatically:

```python
def generic_validate(url: str, method: str = "GET", **kwargs: Any) -> httpx.Response:
    client = get_httpx_client()
    response = client.request(method, url, **kwargs)
    response.raise_for_status()
    return response

```

### Real Usage: Instagram Validator

The Instagram username checker in [`user_scanner/user_scan/social/instagram.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scanner/user_scan/social/instagram.py) demonstrates this pattern:

```python
from ..core.helpers import generic_validate

_PROFILE_URL = "https://www.instagram.com/{user}/?__a=1"

def validate_instagram(user: str):
    url = _PROFILE_URL.format(user=user)
    try:
        response = generic_validate(url)  # ← reuses global httpx.Client

    except httpx.HTTPStatusError as e:
        if e.response.status_code == 404:
            return "available"
        raise
    data = json.loads(response.text)
    return "taken" if data.get("graphql", {}).get("user", {}) else "available"

```

---

## curl_cffi Session Caching: Per-Impersonation-Profile Sessions

For platforms that block generic HTTP clients (e.g., Cloudflare-protected sites like Reddit), User-Scanner uses **curl_cffi** to mimic browser TLS fingerprints. Since creating these sessions involves a warm-up request to obtain clearance cookies, the repository caches them aggressively.

### The Session Cache Design

In [`user_scanner/core/impersonate.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scanner/core/impersonate.py), sessions are stored in `_sessions` with a composite key:

```python
import threading
import curl_cffi

_sessions: dict[tuple[str, str | None], curl_cffi.requests.Session] = {}
_lock = threading.Lock()

```

The cache key combines:
- **impersonate**: The TLS fingerprint string (e.g., `"chrome"`, `"firefox"`)
- **proxy**: The proxy URL string or `None`

This allows different proxy configurations to maintain separate session pools.

### Session Retrieval and Warm-up

The `_get_warm_session` function implements **create-on-demand with warm-up**:

```python
def _get_warm_session(
    impersonate: str,
    proxy: str | None,
    warmup_url: str,
) -> curl_cffi.requests.Session:
    key = (impersonate, proxy)
    with _lock:
        session = _sessions.get(key)
        if session is None:
            session = curl_cffi.requests.Session(impersonate=impersonate, proxy=proxy)
            _sessions[key] = session
        # Warm-up: fetch clearance page once per session

        try:
            session.get(warmup_url, timeout=_timeout())
        except Exception:
            pass  # Allow caller to handle retry logic

    return session

```

The **warm-up request** is crucial—Cloudflare and similar services return challenge pages initially. By performing this once and caching the resulting cookie state, subsequent requests bypass the challenge.

### Public API: `impersonate_validate`

Validators use the high-level wrapper:

```python
def impersonate_validate(
    impersonate: str,
    url: str,
    method: str = "GET",
    **kwargs: Any,
) -> requests.Response:
    from .helpers import get_proxy
    proxy = get_proxy()
    session = _get_warm_session(impersonate, proxy, warmup_url=url)
    return session.request(method, url, timeout=_timeout(), **kwargs)

```

### Real Usage: Reddit Validator

Reddit's bot wall requires impersonation, as shown in [`user_scanner/user_scan/social/reddit.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scanner/user_scan/social/reddit.py):

```python
from ..core.impersonate import impersonate_validate

_BASE_URL = "https://www.reddit.com/user/{user}/about.json"
_WARMUP_URL = "https://www.reddit.com/"

def validate_reddit(user: str):
    url = _BASE_URL.format(user=user)
    response = impersonate_validate(
        impersonate="chrome",
        url=url,
        method="GET",
        warmup_url=_WARMUP_URL,
    )
    data = json.loads(response.text)
    return "taken" if data.get("is_employee") else "available"

```

Note that `warmup_url` is passed as a dedicated parameter—this separates the warm-up target from the actual validation URL, allowing cheaper endpoints to prime the session.

---

## Async Support: Thread Pool Delegation

The email scanning modules are async, but curl_cffi is synchronous. The repository bridges this gap using a **thread pool executor** without breaking session caching:

```python
import asyncio
from concurrent.futures import ThreadPoolExecutor

_executor = ThreadPoolExecutor(max_workers=4)

def _run_in_thread(fn: Callable[..., Any], *args: Any, **kwargs: Any) -> Any:
    loop = asyncio.get_event_loop()
    return loop.run_in_executor(_executor, lambda: fn(*args, **kwargs))


async def impersonate_async(
    impersonate: str,
    url: str,
    method: str = "GET",
    **kwargs: Any,
) -> requests.Response:
    """Async version of impersonate_validate."""
    return await _run_in_thread(impersonate_validate, impersonate, url, method, **kwargs)

```

The same cached session is accessed from worker threads—`threading.Lock` in `_get_warm_session` ensures safety.

---

## Comparison: HTTPX vs. curl_cffi Caching

| Aspect | HTTPX ([`helpers.py`](https://github.com/kaifcodec/user-scanner/blob/main/helpers.py)) | curl_cffi ([`impersonate.py`](https://github.com/kaifcodec/user-scanner/blob/main/impersonate.py)) |
|--------|----------------------|------------------------------|
| **Cache scope** | Single global client | Multiple sessions per `(impersonate, proxy)` key |
| **Key dimension** | None (one client) | Tuple of fingerprint + proxy |
| **Warm-up needed** | No | Yes—primes cookies/TLS state |
| **Thread safety** | `threading.Lock` on initialization | `threading.Lock` on all access |
| **Use case** | Standard sites (Instagram, etc.) | Bot-protected sites (Reddit, Cloudflare) |
| **Async support** | Native `AsyncClient` not cached; sync client used via thread pool | Thread pool wrapper around sync sessions |

Both mechanisms prioritize **connection reuse** to reduce latency under concurrent load.

---

## Summary

- **HTTPX sessions** are cached via a singleton `httpx.Client` in [`user_scaner/core/helpers.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scaner/core/helpers.py), initialized once per process with thread-safe double-checked locking
- **curl_cffi sessions** are cached in a dictionary keyed by `(impersonate, proxy)` tuples in [`user_scaner/core/impersonate.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scaner/core/impersonate.py), with mandatory warm-up requests to seed clearance state
- Both caches use `threading.Lock` for safe access under the `--concurrency` CLI flag
- Async modules access curl_cffi through a thread pool without creating duplicate sessions

---

## Frequently Asked Questions

### Does User-Scanner create a new HTTPX client for every request?

No. The `get_httpx_client()` function in [`user_scaner/core/helpers.py`](https://github.com/kaifcodec/user-scanner/blob/main/user_scaner/core/helpers.py) creates the client once and returns the same instance for all subsequent calls. This is implemented with a thread-safe singleton pattern using `threading.Lock` to handle concurrent initialization safely.

### Why does curl_cffi need a warm-up request?

The warm-up request in `_get_warm_session` obtains **clearance cookies** from services like Cloudflare that present challenge pages to new clients. By caching the session with these cookies, subsequent requests bypass the challenge. Without this, every new session would trigger expensive and potentially blocking verification flows.

### Can I use different proxies with the same impersonation profile?

Yes. The cache key in `_sessions` is `(impersonate, proxy)`, so different proxy configurations automatically create separate session pools. This prevents proxy-specific state (like IP-based rate limits) from leaking between different proxy endpoints.

### Is the async interface for curl_cffi truly non-blocking?

The `impersonate_async` function uses `asyncio.get_event_loop().run_in_executor` with a `ThreadPoolExecutor` to run synchronous curl_cffi calls in background threads. While this doesn't provide true async I/O, it prevents the event loop from blocking while preserving the cached session benefits across async and sync code paths.