DeepWiki-MCP Crawler Retry Mechanism and Exponential Backoff Strategy Explained

The deepwiki-mcp crawler implements a fixed retry limit of 3 attempts with exponential backoff starting at 250ms, doubling the delay after each failure to handle transient network errors gracefully.

The regenrek/deepwiki-mcp repository provides a Model Context Protocol (MCP) server for crawling DeepWiki documentation. Understanding its retry mechanism and exponential backoff strategy is essential for developers troubleshooting network resilience or adapting the crawler for high-latency environments.

Core Retry Configuration in httpCrawler.ts

The retry behavior is hardcoded in src/lib/httpCrawler.ts, where two constants define the resilience policy.

Retry Limit and Backoff Constants

At lines 11-12, the crawler declares:

• RETRY_LIMIT – Set to 3, defining the maximum number of attempts before marking a URL as failed.
• BACKOFF_BASE_MS – Set to 250 milliseconds, serving as the initial delay duration.

These values are not exposed through the public API, ensuring consistent behavior across all crawling sessions unless the source is modified directly.

Exponential Backoff Algorithm Implementation

The crawler applies exponential backoff inside the request loop (lines 36-84) to prevent overwhelming target servers during temporary outages.

Backoff Calculation Logic

After each failed attempt, the crawler calculates the delay using:

BACKOFF_BASE_MS * 2 ** (retries - 1)

This produces the following delay sequence:

• Attempt 1 failure: 250ms delay
• Attempt 2 failure: 500ms delay
• Attempt 3 failure: 1000ms delay

Request Loop Implementation

The following TypeScript excerpt from src/lib/httpCrawler.ts demonstrates the retry logic:

while (true) {
  try {
    // … fetch logic …
    return;                         // success → exit
  }
  catch (err) {
    if (retries < RETRY_LIMIT) {
      retries++;
      await setTimeout(BACKOFF_BASE_MS * 2 ** (retries - 1));
      continue;                     // retry
    }
    errors.push({ path: key, reason: String(err) });
    return;                         // give‑up
  }
}

The loop continues until either the fetch succeeds or the retry limit is exhausted, at which point the error is recorded and the crawler proceeds to the next URL.

Practical Usage and Error Handling

While the retry mechanism operates internally, understanding its behavior helps interpret crawler output and debug network issues.

Basic Crawler Usage

The retry logic triggers automatically when using the crawl function:

import { crawl } from './src/lib/httpCrawler';
import { URL } from 'node:url';

async function run() {
  const result = await crawl({
    root: new URL('https://deepwiki.com/owner/repo'),
    maxDepth: 1,
    emit: (e) => console.log('Progress:', e),
    verbose: true,
  });

  console.log('Fetched pages:', Object.keys(result.html));
  console.log('Errors:', result.errors);
}
run();

Customization Limitations

The RETRY_LIMIT and BACKOFF_BASE_MS constants are defined at the module level in src/lib/httpCrawler.ts and are not exposed through configuration parameters. To adjust retry behavior, developers must modify these constants directly in the source code before compilation.

Summary

• The retry mechanism in regenrek/deepwiki-mcp limits failed requests to 3 attempts defined by RETRY_LIMIT in src/lib/httpCrawler.ts.
• Exponential backoff starts at 250ms (BACKOFF_BASE_MS) and doubles with each retry, creating delays of 250ms, 500ms, and 1000ms.
• The implementation resides in the request loop (lines 36-84) of src/lib/httpCrawler.ts, using setTimeout with the calculation BACKOFF_BASE_MS * 2 ** (retries - 1).
• These values are hardcoded constants and cannot be configured through the public API without modifying the source.

Frequently Asked Questions

How many retry attempts does the deepwiki-mcp crawler make?

The crawler attempts each request up to 3 times before marking it as failed. This is controlled by the RETRY_LIMIT constant defined at line 11 of src/lib/httpCrawler.ts.

What is the maximum delay between retry attempts?

The maximum delay is 1000 milliseconds (1 second). The crawler uses exponential backoff starting at 250ms, so the delays progress through 250ms, 500ms, and finally 1000ms on the third and final retry attempt.

Can I customize the retry limit or backoff timing?

No, these parameters are hardcoded as constants (RETRY_LIMIT and BACKOFF_BASE_MS) in src/lib/httpCrawler.ts and are not exposed through the public crawl() function API. To change these values, you must modify the source code directly and recompile the project.

Where is the retry logic implemented in the codebase?

The retry mechanism and exponential backoff algorithm are implemented in the request loop located at lines 36-84 of src/lib/httpCrawler.ts. This section contains the while loop that manages fetch attempts, calculates backoff delays using setTimeout, and tracks the retry counter against RETRY_LIMIT.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}