How ProxyManager Manages Request Concurrency Limits with Credentials in WeChat Article Exporter

ProxyManager delegates concurrency control to BaseDownloader while managing per-proxy health states, allowing credentialed requests to use lower concurrency limits to avoid rate limiting.

The wechat-article-exporter project separates download orchestration from proxy lifecycle management to handle high-volume scraping safely. While BaseDownloader enforces global request concurrency limits through an internal queue, ProxyManager tracks individual proxy health and rotates endpoints to prevent credential-related rate limits from triggering IP bans. This architecture ensures that sensitive API calls requiring authentication receive stricter throttling than standard HTML fetches.

Concurrency Control Architecture

Parallel request limits are calculated and enforced inside BaseDownloader, not within ProxyManager. When instantiating a downloader, the concurrency option defaults to a value proportional to the number of available proxies.

Calculating Default Concurrency

In utils/download/BaseDownloader.ts, the constructor sets options using bestConcurrencyCount:

this.options = {
  concurrency: options.concurrency ?? bestConcurrencyCount(proxies.length),
  timeout: …,
  maxRetries: …,
  cooldownPeriod: …,
  maxFailures: …,
};

The bestConcurrencyCount function (defined in utils/index.ts) returns a concurrency value scaled to the proxy pool size. This ensures the downloader does not overwhelm the available infrastructure by default.

Enforcing Limits with p-queue

The downloader uses an internal p-queue to schedule tasks. When processing requests that require credentials—such as page-view or comment APIs—the caller can override the default with a conservative value to respect stricter rate limits:

// High concurrency for public HTML (no credentials)
const htmlDownloader = new BaseDownloader(urlList, { concurrency: 20 });

// Low concurrency for credentialed endpoints
const viewDownloader = new BaseDownloader(viewUrlList, { 
  concurrency: 5  // Respect credential rate-limit
});

Proxy Health and Selection

While BaseDownloader decides how many requests run in parallel, ProxyManager decides which proxy handles each request. It maintains a per-proxy status map tracking failures, lastUsed, cooldown, and totalUse.

Core ProxyManager Methods

Located in utils/download/ProxyManager.ts, the class exposes four critical methods:

• getBestProxy() – Returns the proxy with the fewest failures that is not currently in cooldown, updating lastUsed and totalUse timestamps.
• recordFailure(proxy) – Increments the failure counter; when failures ≥ maxFailures, sets cooldown = true to pause usage.
• recordSuccess(proxy) – Resets the failure counter and clears the cooldown flag.
• resetAndGetProxy() – When all proxies are cooling down, refreshes the oldest-used proxy (resetting failures) and returns it.

Failure Handling Integration

BaseDownloader invokes these methods within its lifecycle hooks. After a failure, it records the incident:

// utils/download/BaseDownloader.ts
protected async handleDownloadFailure(proxy: string, url: string, attempt: number, error: any) {
  this.proxyManager.recordFailure(proxy);
  // … retry logic …
}

On success, it clears the failure state:

this.proxyManager.recordSuccess(proxy);

Handling Credentialed Requests

Requests requiring authentication follow the same pipeline but with additional safeguards. Credentials stored in auto-detect-credentials:credentials local storage are injected only when withCredential is true.

Reduced Concurrency for API Endpoints

Because credentialed endpoints enforce stricter rate limits, the typical pattern instantiates a dedicated downloader with reduced concurrency:

const credentialedDownloader = new BaseDownloader(apiUrls, {
  concurrency: 5,  // Prevents hitting credential-based rate limits
});

While this downloader runs, ProxyManager continues rotating proxies normally. If a proxy fails repeatedly, it enters cooldown, and the next-best proxy takes its place in the concurrency pool.

Proxy Rotation During Execution

Even with low concurrency, the proxy selection remains dynamic. Each concurrent slot queries getBestProxy() independently, ensuring that a failing credential request does not stall the entire queue—the ProxyManager simply routes the retry through a different endpoint.

Practical Implementation Example

The following pattern demonstrates the separation of concerns when mixing public and private data fetching:

import { BaseDownloader } from '@/utils/download/BaseDownloader';

// 1. Download public HTML with high throughput
const htmlDownloader = new BaseDownloader(htmlUrls, { 
  concurrency: 20 
});
htmlDownloader.on('finished', () => console.log('HTML batch complete'));
htmlDownloader.start();

// 2. Download page-view metrics (requires credentials)
const viewDownloader = new BaseDownloader(viewUrls, { 
  concurrency: 5  // Safe for credentialed APIs
});
viewDownloader.start();

// 3. Monitor proxy health during operation
const status = htmlDownloader.getStatus();
console.log('Proxy health:', status.proxy);

Summary

• Concurrency limits are calculated in BaseDownloader and enforced by an internal p-queue, defaulting to values proportional to the proxy pool size via bestConcurrencyCount in utils/index.ts.
• Proxy health is tracked by ProxyManager through per-proxy counters (failures, lastUsed, cooldown) and methods like recordFailure() and getBestProxy().
• Credential protection is achieved by instantiating downloaders with reduced concurrency values (typically 5) for API endpoints, while ProxyManager continues rotating healthy proxies.
• Failover logic automatically cools down failing proxies and resets them only when the entire pool is exhausted via resetAndGetProxy(), preventing credential-triggered IP bans from halting operations.

Frequently Asked Questions

How does ProxyManager know when to stop using a failing proxy?

ProxyManager increments an internal failure counter via recordFailure() each time BaseDownloader reports an error. Once the failure count reaches maxFailures, the proxy is marked with cooldown = true and excluded from getBestProxy() selections until recordSuccess() clears the state or resetAndGetProxy() force-resets it.

Why is concurrency controlled by BaseDownloader instead of ProxyManager?

Separating concerns allows ProxyManager to focus solely on endpoint health and selection, while BaseDownloader manages the global task queue. This architecture lets developers specify different concurrency limits for credentialed versus public requests without modifying proxy rotation logic.

What happens if all proxies enter cooldown simultaneously?

When every proxy is cooling down, ProxyManager.resetAndGetProxy() identifies the oldest-used proxy, resets its failure counter, and returns it immediately. This emergency release valve prevents deadlock while still penalizing consistently failing endpoints.

How are credentials injected into requests requiring authentication?

When withCredential is enabled, the downloader retrieves session data from the auto-detect-credentials:credentials local storage key and appends it to request headers. The same ProxyManager selection logic applies, but the reduced concurrency limit prevents overwhelming the authenticated API endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}