# DeepWiki-MCP Crawler Retry Mechanism and Exponential Backoff Strategy Explained

> Discover the deepwiki-mcp crawler retry mechanism. Learn how exponential backoff handles transient errors with a 3-attempt limit and 250ms starting delay.

- Repository: [Kevin Kern/deepwiki-mcp](https://github.com/regenrek/deepwiki-mcp)
- Tags: deep-dive
- Published: 2026-02-16

---

**The deepwiki-mcp crawler implements a fixed retry limit of 3 attempts with exponential backoff starting at 250ms, doubling the delay after each failure to handle transient network errors gracefully.**

The `regenrek/deepwiki-mcp` repository provides a Model Context Protocol (MCP) server for crawling DeepWiki documentation. Understanding its **retry mechanism and exponential backoff strategy** is essential for developers troubleshooting network resilience or adapting the crawler for high-latency environments.

## Core Retry Configuration in httpCrawler.ts

The retry behavior is hardcoded in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts), where two constants define the resilience policy.

### Retry Limit and Backoff Constants

At lines 11-12, the crawler declares:

- **`RETRY_LIMIT`** – Set to **3**, defining the maximum number of attempts before marking a URL as failed.
- **`BACKOFF_BASE_MS`** – Set to **250** milliseconds, serving as the initial delay duration.

These values are not exposed through the public API, ensuring consistent behavior across all crawling sessions unless the source is modified directly.

## Exponential Backoff Algorithm Implementation

The crawler applies exponential backoff inside the request loop (lines 36-84) to prevent overwhelming target servers during temporary outages.

### Backoff Calculation Logic

After each failed attempt, the crawler calculates the delay using:

```typescript
BACKOFF_BASE_MS * 2 ** (retries - 1)

```

This produces the following delay sequence:
- **Attempt 1 failure:** 250ms delay
- **Attempt 2 failure:** 500ms delay  
- **Attempt 3 failure:** 1000ms delay

### Request Loop Implementation

The following TypeScript excerpt from [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) demonstrates the retry logic:

```typescript
while (true) {
  try {
    // … fetch logic …
    return;                         // success → exit
  }
  catch (err) {
    if (retries < RETRY_LIMIT) {
      retries++;
      await setTimeout(BACKOFF_BASE_MS * 2 ** (retries - 1));
      continue;                     // retry
    }
    errors.push({ path: key, reason: String(err) });
    return;                         // give‑up
  }
}

```

The loop continues until either the fetch succeeds or the retry limit is exhausted, at which point the error is recorded and the crawler proceeds to the next URL.

## Practical Usage and Error Handling

While the retry mechanism operates internally, understanding its behavior helps interpret crawler output and debug network issues.

### Basic Crawler Usage

The retry logic triggers automatically when using the `crawl` function:

```typescript
import { crawl } from './src/lib/httpCrawler';
import { URL } from 'node:url';

async function run() {
  const result = await crawl({
    root: new URL('https://deepwiki.com/owner/repo'),
    maxDepth: 1,
    emit: (e) => console.log('Progress:', e),
    verbose: true,
  });

  console.log('Fetched pages:', Object.keys(result.html));
  console.log('Errors:', result.errors);
}
run();

```

### Customization Limitations

The `RETRY_LIMIT` and `BACKOFF_BASE_MS` constants are defined at the module level in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) and are not exposed through configuration parameters. To adjust retry behavior, developers must modify these constants directly in the source code before compilation.

## Summary

- The **retry mechanism** in `regenrek/deepwiki-mcp` limits failed requests to **3 attempts** defined by `RETRY_LIMIT` in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts).
- **Exponential backoff** starts at **250ms** (`BACKOFF_BASE_MS`) and doubles with each retry, creating delays of 250ms, 500ms, and 1000ms.
- The implementation resides in the request loop (lines 36-84) of [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts), using `setTimeout` with the calculation `BACKOFF_BASE_MS * 2 ** (retries - 1)`.
- These values are hardcoded constants and cannot be configured through the public API without modifying the source.

## Frequently Asked Questions

### How many retry attempts does the deepwiki-mcp crawler make?

The crawler attempts each request up to **3 times** before marking it as failed. This is controlled by the `RETRY_LIMIT` constant defined at line 11 of [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts).

### What is the maximum delay between retry attempts?

The maximum delay is **1000 milliseconds** (1 second). The crawler uses exponential backoff starting at 250ms, so the delays progress through 250ms, 500ms, and finally 1000ms on the third and final retry attempt.

### Can I customize the retry limit or backoff timing?

No, these parameters are hardcoded as constants (`RETRY_LIMIT` and `BACKOFF_BASE_MS`) in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) and are not exposed through the public `crawl()` function API. To change these values, you must modify the source code directly and recompile the project.

### Where is the retry logic implemented in the codebase?

The retry mechanism and exponential backoff algorithm are implemented in the request loop located at **lines 36-84** of [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts). This section contains the `while` loop that manages fetch attempts, calculates backoff delays using `setTimeout`, and tracks the retry counter against `RETRY_LIMIT`.