# How the wechat-article-exporter Implements a Concurrent Download Queue with p-queue

> Learn how wechat-article-exporter uses p-queue to implement a concurrent download queue. Dynamically set concurrency limits with proxies to avoid rate limiting and overload.

- Repository: [公众号文章工具箱/wechat-article-exporter](https://github.com/wechat-article/wechat-article-exporter)
- Tags: internals
- Published: 2026-05-26

---

**The wechat-article-exporter project manages bulk article downloads by combining a proxy pool with the p-queue library, dynamically setting concurrency limits to match the number of available proxies to prevent rate limiting and overload.**

The `wechat-article/wechat-article-exporter` repository provides a robust solution for batch downloading WeChat articles while respecting network constraints. By implementing a **concurrent download queue** using the **p-queue** library alongside a custom proxy pool defined in [`utils/pool.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/pool.ts), the system ensures optimal throughput without exceeding proxy capacity.

## Proxy Pool Architecture in [`utils/pool.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/pool.ts)

The foundation of the concurrency system rests on the `ProxyPool` class. This pool manages both public proxies from [`config/public-proxy.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/config/public-proxy.ts) and private proxies stored in `localStorage`. Each proxy object maintains:

- A **`busy`** flag indicating active use
- Usage counters tracking request volume
- A cooldown mechanism to prevent overuse

Before any downloads begin, the `downloads` helper initializes the pool with optional private proxy configurations:

```ts
const privateProxy: string[] = [];
try {
  const proxy = JSON.parse(window.localStorage.getItem('wechat-proxy')!);
  if (Array.isArray(proxy) && proxy.length > 0) privateProxy.push(...proxy);
} catch {}
pool.init(privateProxy);

```

## Configuring p-queue Concurrency Limits

The `downloads` function creates a `PQueue` instance where the `concurrency` option dynamically matches the available proxy count. According to the source code in [`utils/pool.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/pool.ts):

```ts
const queue = new PQueue({ concurrency: pool.proxies.length });

```

This direct correlation ensures the number of simultaneous downloads never exceeds the number of usable proxies. When private proxies are added via `localStorage`, the concurrency limit automatically increases to utilize these additional resources.

## Task Scheduling and Execution

Individual download tasks wrap the internal `download` helper and enqueue via `queue.add`. The implementation maps each resource to a queue task:

```ts
const tasks = resources.map(resource =>
  queue.add(() => download<T>(resource, downloadFn, useProxy))
);
await Promise.all(tasks);

```

The `queue.add` method returns a promise for each task. Collecting these into a `tasks` array and awaiting `Promise.all(tasks)` ensures the function only returns after every resource processes (or fails after internal retries).

## Proxy Coordination and Backpressure

The system implements intelligent backpressure through the proxy pool's `releaseProxy` method. When a download completes in [`utils/download/Downloader.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/download/Downloader.ts), the proxy marks itself as available, triggering p-queue to automatically schedule the next waiting task.

This coordination guarantees three critical behaviors:
- **One download per proxy**: Concurrency strictly matches proxy availability
- **Automatic scheduling**: New tasks begin immediately when proxies free up
- **Rate limit protection**: No single proxy handles multiple simultaneous requests

The [`Downloader.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/Downloader.ts) component handles the actual HTTP logic and retries, while the queue in [`utils/pool.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/utils/pool.ts) strictly manages concurrency timing.

## Summary

The concurrent download queue implementation in wechat-article-exporter demonstrates a robust pattern for resource-constrained batch processing:

- **Proxy-aware limits**: The `PQueue` concurrency equals `pool.proxies.length` to match network capacity
- **Dynamic initialization**: Supports hybrid proxy configurations mixing public and private sources
- **Promise-based execution**: `Promise.all(tasks)` ensures complete batch completion before proceeding
- **Automatic backpressure**: Proxy release triggers immediate task scheduling without manual intervention

## Frequently Asked Questions

### Why is the concurrency limit tied to the number of proxies?

Setting `concurrency: pool.proxies.length` prevents any single proxy from handling multiple simultaneous requests. This hard limit stops individual proxies from being overwhelmed, which would trigger rate limits or IP bans from the target servers.

### How does the system handle private versus public proxies?

The `downloads` function checks `localStorage` for private proxy configurations before initializing the pool. These private proxies join the `PUBLIC_PROXY_LIST` from [`config/public-proxy.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/config/public-proxy.ts), expanding the total proxy count and thereby increasing the concurrency limit while maintaining the one-task-per-proxy rule.

### What happens when a download task fails?

Individual task failures are handled within the `download` helper called by `queue.add`. The [`Downloader.ts`](https://github.com/wechat-article/wechat-article-exporter/blob/main/Downloader.ts) implementation manages retry logic and error handling. Failed tasks still resolve their promises (potentially with error values), allowing `Promise.all` to complete without blocking the entire batch.

### Can this pattern work without proxies?

Yes. When `useProxy` is set to `false` or the proxy pool is disabled, the system can still utilize p-queue for concurrency control, though the implementation specifically optimizes for proxy-aware scenarios where limiting concurrent connections per IP address is critical.