How the wechat-article-exporter Implements a Concurrent Download Queue with p-queue

The wechat-article-exporter project manages bulk article downloads by combining a proxy pool with the p-queue library, dynamically setting concurrency limits to match the number of available proxies to prevent rate limiting and overload.

The wechat-article/wechat-article-exporter repository provides a robust solution for batch downloading WeChat articles while respecting network constraints. By implementing a concurrent download queue using the p-queue library alongside a custom proxy pool defined in utils/pool.ts, the system ensures optimal throughput without exceeding proxy capacity.

Proxy Pool Architecture in utils/pool.ts

The foundation of the concurrency system rests on the ProxyPool class. This pool manages both public proxies from config/public-proxy.ts and private proxies stored in localStorage. Each proxy object maintains:

• A busy flag indicating active use
• Usage counters tracking request volume
• A cooldown mechanism to prevent overuse

Before any downloads begin, the downloads helper initializes the pool with optional private proxy configurations:

const privateProxy: string[] = [];
try {
  const proxy = JSON.parse(window.localStorage.getItem('wechat-proxy')!);
  if (Array.isArray(proxy) && proxy.length > 0) privateProxy.push(...proxy);
} catch {}
pool.init(privateProxy);

Configuring p-queue Concurrency Limits

The downloads function creates a PQueue instance where the concurrency option dynamically matches the available proxy count. According to the source code in utils/pool.ts:

const queue = new PQueue({ concurrency: pool.proxies.length });

This direct correlation ensures the number of simultaneous downloads never exceeds the number of usable proxies. When private proxies are added via localStorage, the concurrency limit automatically increases to utilize these additional resources.

Task Scheduling and Execution

Individual download tasks wrap the internal download helper and enqueue via queue.add. The implementation maps each resource to a queue task:

const tasks = resources.map(resource =>
  queue.add(() => download<T>(resource, downloadFn, useProxy))
);
await Promise.all(tasks);

The queue.add method returns a promise for each task. Collecting these into a tasks array and awaiting Promise.all(tasks) ensures the function only returns after every resource processes (or fails after internal retries).

Proxy Coordination and Backpressure

The system implements intelligent backpressure through the proxy pool's releaseProxy method. When a download completes in utils/download/Downloader.ts, the proxy marks itself as available, triggering p-queue to automatically schedule the next waiting task.

This coordination guarantees three critical behaviors:

• One download per proxy: Concurrency strictly matches proxy availability
• Automatic scheduling: New tasks begin immediately when proxies free up
• Rate limit protection: No single proxy handles multiple simultaneous requests

The Downloader.ts component handles the actual HTTP logic and retries, while the queue in utils/pool.ts strictly manages concurrency timing.

Summary

The concurrent download queue implementation in wechat-article-exporter demonstrates a robust pattern for resource-constrained batch processing:

• Proxy-aware limits: The PQueue concurrency equals pool.proxies.length to match network capacity
• Dynamic initialization: Supports hybrid proxy configurations mixing public and private sources
• Promise-based execution: Promise.all(tasks) ensures complete batch completion before proceeding
• Automatic backpressure: Proxy release triggers immediate task scheduling without manual intervention

Frequently Asked Questions

Why is the concurrency limit tied to the number of proxies?

Setting concurrency: pool.proxies.length prevents any single proxy from handling multiple simultaneous requests. This hard limit stops individual proxies from being overwhelmed, which would trigger rate limits or IP bans from the target servers.

How does the system handle private versus public proxies?

The downloads function checks localStorage for private proxy configurations before initializing the pool. These private proxies join the PUBLIC_PROXY_LIST from config/public-proxy.ts, expanding the total proxy count and thereby increasing the concurrency limit while maintaining the one-task-per-proxy rule.

What happens when a download task fails?

Individual task failures are handled within the download helper called by queue.add. The Downloader.ts implementation manages retry logic and error handling. Failed tasks still resolve their promises (potentially with error values), allowing Promise.all to complete without blocking the entire batch.

Can this pattern work without proxies?

Yes. When useProxy is set to false or the proxy pool is disabled, the system can still utilize p-queue for concurrency control, though the implementation specifically optimizes for proxy-aware scenarios where limiting concurrent connections per IP address is critical.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}