# How the DeepWiki-MCP Crawler Handles Partial Failures and Aggregates Error Responses

> Discover how the deepwiki-mcp crawler expertly handles partial failures and aggregates errors. Learn about its isolated processing, retries, and structured error reporting.

- Repository: [Kevin Kern/deepwiki-mcp](https://github.com/regenrek/deepwiki-mcp)
- Tags: internals
- Published: 2026-02-16

---

**The DeepWiki-MCP crawler processes each URL in isolation using a promise queue, implements exponential backoff retries up to three times, and aggregates all failures into a structured `errors` array returned alongside successful results.**

The `regenrek/deepwiki-mcp` repository implements a fault-tolerant web crawler designed to continue operation even when individual pages fail to load. Understanding how this system handles partial failures is essential for building resilient documentation pipelines that process large URL sets without terminating on isolated network errors.

## Isolated Per-URL Execution with PQueue

The crawler leverages **PQueue** (promise queue) to ensure that failures remain isolated to individual URLs. Each fetch operation executes as an independent task within the queue, meaning a network timeout or HTTP error in one task does not propagate to others or halt the overall crawl process.

This architecture is implemented in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) within the main `crawl` function, where the queue processes URLs concurrently while maintaining strict error boundaries between tasks.

## Retry Logic with Exponential Backoff

When a fetch fails, the crawler does not immediately record it as a permanent failure. Instead, it enters a retry loop with exponential backoff to handle transient network issues.

### Configurable Retry Limits

The system uses two constants defined in the crawler:

- `RETRY_LIMIT`: Defaults to **3** attempts per URL
- `BACKOFF_BASE_MS`: Base milliseconds for calculating delay intervals

### Backoff Strategy Implementation

The retry logic resides inside the `while (true)` loop within each queue task. After a failure, the crawler calculates the delay using exponential backoff:

```typescript
// src/lib/httpCrawler.ts#L78-L84
if (retries < RETRY_LIMIT) {
  retries++;
  await setTimeout(BACKOFF_BASE_MS * 2 ** (retries - 1));
  continue;
}

```

This approach waits progressively longer between attempts (e.g., 1s, 2s, 4s) to avoid overwhelming struggling servers while maximizing the chance of recovery from temporary outages.

## Error Capture and Aggregation

Once the retry limit is exhausted, the crawler transitions from recovery to recording mode, ensuring no error data is lost while maintaining workflow continuity.

### Local Error Collection

Rather than throwing exceptions that would disrupt the queue, the crawler pushes failure details into a local `errors` array:

```typescript
// src/lib/httpCrawler.ts#L84-L86
errors.push({ path: key, reason: String(err) })

```

Each entry contains the `path` (URL identifier) and the stringified error reason, creating a structured log of all failures that occurred during the crawl session.

### CrawlResult Structure

After the queue drains completely, the `crawl` function returns a comprehensive `CrawlResult` object that unifies successes and failures:

```typescript
// src/lib/httpCrawler.ts#L94-L100
return { html, errors, bytes: totalBytes, elapsedMs }

```

This structure includes:
- `html`: Object mapping URLs to fetched content
- `errors`: Array of all failed URL records
- `bytes`: Total data transferred
- `elapsedMs`: Total execution time

## Real-Time Progress Monitoring

The crawler exposes failure states through **ProgressEvent** emissions, allowing calling code to monitor partial failures in real time without interrupting the crawl.

Each request emits events containing the current `retries` count, enabling live dashboards or logs to display which URLs are struggling and how many retry attempts they have consumed. This mechanism is defined in [`src/types.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/types.ts) and utilized throughout the crawler implementation.

## Practical Implementation Example

The following example demonstrates initiating a crawl, monitoring progress events, and handling the aggregated error list upon completion:

```typescript
import { crawl } from '@/lib/httpCrawler';
import { URL } from 'node:url';

// Start a crawl with a depth limit of 2
const result = await crawl({
  root: new URL('https://example.com/'),
  maxDepth: 2,
  // Simple logger that prints progress
  emit: (e) => console.log(`[${e.type}] ${e.url} – retries:${e.retries}`),
  verbose: true,
});

console.log('Fetched pages:', Object.keys(result.html).length);
console.log('Failed pages:', result.errors.length);
result.errors.forEach(err => {
  console.error(`❌ ${err.path} → ${err.reason}`);
});

```

This implementation showcases the fault-tolerant design: even if specific pages fail after three retry attempts, the crawl continues, and all failures are accessible via `result.errors` for post-processing or reporting.

## Summary

The DeepWiki-MCP crawler handles partial failures through a multi-layered resilience strategy:

- **Isolated execution** via PQueue prevents single failures from aborting the entire crawl
- **Exponential backoff retries** (up to 3 attempts) recover transient network issues
- **Structured error aggregation** collects all failures into a returnable `errors` array
- **Real-time progress events** expose retry states without interrupting execution
- **Comprehensive result objects** unify successful fetches and error logs in `CrawlResult`

## Frequently Asked Questions

### What happens when a URL fails permanently after all retries?

When a URL exhausts the `RETRY_LIMIT` (default 3), the crawler captures the error in the local `errors` array with the path and error message, then continues processing remaining URLs. The failure is included in the final `CrawlResult` returned by the `crawl` function in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts).

### How does the crawler prevent one failure from stopping the entire job?

The crawler uses **PQueue** to execute each URL fetch as an independent task. Errors within individual tasks are caught and handled locally without propagating to the queue controller, ensuring that network timeouts or HTTP errors remain isolated to specific URLs while the broader crawl continues uninterrupted.

### Can I adjust the retry behavior?

Yes, the retry logic uses configurable constants `RETRY_LIMIT` and `BACKOFF_BASE_MS` defined in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts). While the current implementation uses default values (3 retries with exponential backoff), these constants can be modified in the source to adjust the maximum attempts or the base delay interval between retries.

### How do I access the list of failed URLs after crawling?

After the `crawl` function completes, access the `errors` property on the returned `CrawlResult` object. This array contains objects with `path` (the failed URL) and `reason` (the error message) for every URL that failed after exhausting all retry attempts, allowing you to log, report, or reprocess failed pages as needed.