# How ProxyManager Manages Request Concurrency Limits with Credentials in WeChat Article Exporter

> Learn how ProxyManager manages request concurrency limits with credentials by delegating control to BaseDownloader while using lower limits for credentialed requests to prevent rate limiting.

- Repository: [公众号文章工具箱/wechat-article-exporter](https://github.com/wechat-article/wechat-article-exporter)
- Tags: how-to-guide
- Published: 2026-05-26

---

**ProxyManager delegates concurrency control to BaseDownloader while managing per-proxy health states, allowing credentialed requests to use lower concurrency limits to avoid rate limiting.**

The `wechat-article-exporter` project separates download orchestration from proxy lifecycle management to handle high-volume scraping safely. While `BaseDownloader` enforces global request concurrency limits through an internal queue, `ProxyManager` tracks individual proxy health and rotates endpoints to prevent credential-related rate limits from triggering IP bans. This architecture ensures that sensitive API calls requiring authentication receive stricter throttling than standard HTML fetches.

## Concurrency Control Architecture

Parallel request limits are calculated and enforced inside `BaseDownloader`, not within `ProxyManager`. When instantiating a downloader, the `concurrency` option defaults to a value proportional to the number of available proxies.

### Calculating Default Concurrency

In [`utils/download/BaseDownloader.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/download/BaseDownloader.ts), the constructor sets options using `bestConcurrencyCount`:

```typescript
this.options = {
  concurrency: options.concurrency ?? bestConcurrencyCount(proxies.length),
  timeout: …,
  maxRetries: …,
  cooldownPeriod: …,
  maxFailures: …,
};

```

The `bestConcurrencyCount` function (defined in [`utils/index.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/index.ts)) returns a concurrency value scaled to the proxy pool size. This ensures the downloader does not overwhelm the available infrastructure by default.

### Enforcing Limits with p-queue

The downloader uses an internal `p-queue` to schedule tasks. When processing requests that require credentials—such as page-view or comment APIs—the caller can override the default with a conservative value to respect stricter rate limits:

```typescript
// High concurrency for public HTML (no credentials)
const htmlDownloader = new BaseDownloader(urlList, { concurrency: 20 });

// Low concurrency for credentialed endpoints
const viewDownloader = new BaseDownloader(viewUrlList, { 
  concurrency: 5  // Respect credential rate-limit
});

```

## Proxy Health and Selection

While `BaseDownloader` decides **how many** requests run in parallel, `ProxyManager` decides **which proxy** handles each request. It maintains a per-proxy status map tracking `failures`, `lastUsed`, `cooldown`, and `totalUse`.

### Core ProxyManager Methods

Located in [`utils/download/ProxyManager.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/download/ProxyManager.ts), the class exposes four critical methods:

- **`getBestProxy()`** – Returns the proxy with the fewest failures that is not currently in cooldown, updating `lastUsed` and `totalUse` timestamps.
- **`recordFailure(proxy)`** – Increments the failure counter; when `failures ≥ maxFailures`, sets `cooldown = true` to pause usage.
- **`recordSuccess(proxy)`** – Resets the failure counter and clears the cooldown flag.
- **`resetAndGetProxy()`** – When all proxies are cooling down, refreshes the oldest-used proxy (resetting failures) and returns it.

### Failure Handling Integration

`BaseDownloader` invokes these methods within its lifecycle hooks. After a failure, it records the incident:

```typescript
// utils/download/BaseDownloader.ts
protected async handleDownloadFailure(proxy: string, url: string, attempt: number, error: any) {
  this.proxyManager.recordFailure(proxy);
  // … retry logic …
}

```

On success, it clears the failure state:

```typescript
this.proxyManager.recordSuccess(proxy);

```

## Handling Credentialed Requests

Requests requiring authentication follow the same pipeline but with additional safeguards. Credentials stored in `auto-detect-credentials:credentials` local storage are injected only when `withCredential` is `true`.

### Reduced Concurrency for API Endpoints

Because credentialed endpoints enforce stricter rate limits, the typical pattern instantiates a dedicated downloader with reduced concurrency:

```typescript
const credentialedDownloader = new BaseDownloader(apiUrls, {
  concurrency: 5,  // Prevents hitting credential-based rate limits
});

```

While this downloader runs, `ProxyManager` continues rotating proxies normally. If a proxy fails repeatedly, it enters cooldown, and the next-best proxy takes its place in the concurrency pool.

### Proxy Rotation During Execution

Even with low concurrency, the proxy selection remains dynamic. Each concurrent slot queries `getBestProxy()` independently, ensuring that a failing credential request does not stall the entire queue—the `ProxyManager` simply routes the retry through a different endpoint.

## Practical Implementation Example

The following pattern demonstrates the separation of concerns when mixing public and private data fetching:

```typescript
import { BaseDownloader } from '@/utils/download/BaseDownloader';

// 1. Download public HTML with high throughput
const htmlDownloader = new BaseDownloader(htmlUrls, { 
  concurrency: 20 
});
htmlDownloader.on('finished', () => console.log('HTML batch complete'));
htmlDownloader.start();

// 2. Download page-view metrics (requires credentials)
const viewDownloader = new BaseDownloader(viewUrls, { 
  concurrency: 5  // Safe for credentialed APIs
});
viewDownloader.start();

// 3. Monitor proxy health during operation
const status = htmlDownloader.getStatus();
console.log('Proxy health:', status.proxy);

```

## Summary

- **Concurrency limits** are calculated in `BaseDownloader` and enforced by an internal `p-queue`, defaulting to values proportional to the proxy pool size via `bestConcurrencyCount` in [`utils/index.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/index.ts).
- **Proxy health** is tracked by `ProxyManager` through per-proxy counters (`failures`, `lastUsed`, `cooldown`) and methods like `recordFailure()` and `getBestProxy()`.
- **Credential protection** is achieved by instantiating downloaders with reduced `concurrency` values (typically 5) for API endpoints, while `ProxyManager` continues rotating healthy proxies.
- **Failover logic** automatically cools down failing proxies and resets them only when the entire pool is exhausted via `resetAndGetProxy()`, preventing credential-triggered IP bans from halting operations.

## Frequently Asked Questions

### How does ProxyManager know when to stop using a failing proxy?

`ProxyManager` increments an internal failure counter via `recordFailure()` each time `BaseDownloader` reports an error. Once the failure count reaches `maxFailures`, the proxy is marked with `cooldown = true` and excluded from `getBestProxy()` selections until `recordSuccess()` clears the state or `resetAndGetProxy()` force-resets it.

### Why is concurrency controlled by BaseDownloader instead of ProxyManager?

Separating concerns allows `ProxyManager` to focus solely on endpoint health and selection, while `BaseDownloader` manages the global task queue. This architecture lets developers specify different concurrency limits for credentialed versus public requests without modifying proxy rotation logic.

### What happens if all proxies enter cooldown simultaneously?

When every proxy is cooling down, `ProxyManager.resetAndGetProxy()` identifies the oldest-used proxy, resets its failure counter, and returns it immediately. This emergency release valve prevents deadlock while still penalizing consistently failing endpoints.

### How are credentials injected into requests requiring authentication?

When `withCredential` is enabled, the downloader retrieves session data from the `auto-detect-credentials:credentials` local storage key and appends it to request headers. The same `ProxyManager` selection logic applies, but the reduced concurrency limit prevents overwhelming the authenticated API endpoints.