Environment Variables That Control Concurrency, Timeout, and Retry Behavior in deepwiki-mcp

The deepwiki-mcp crawler respects four environment variables—DEEPWIKI_MAX_CONCURRENCY, DEEPWIKI_REQUEST_TIMEOUT, DEEPWIKI_MAX_RETRIES, and DEEPWIKI_RETRY_DELAY—which default to 5 concurrent requests, 30000ms timeout, 3 retry attempts, and 250ms base delay respectively.

The regenrek/deepwiki-mcp repository provides a Model Context Protocol (MCP) server for crawling and analyzing documentation sources. Understanding which environment variables control concurrency, timeout, and retry behavior in deepwiki-mcp enables precise tuning of HTTP request handling to prevent rate limiting and gracefully handle transient network failures.

Core Environment Variables for HTTP Crawling

The crawler configuration relies on four optional environment variables defined in src/lib/httpCrawler.ts. When omitted, the system applies conservative defaults suitable for most documentation crawling scenarios.

Variable	Source Code Reference	Default	Purpose
DEEPWIKI_MAX_CONCURRENCY	process.env.DEEPWIKI_CONCURRENCY in src/lib/httpCrawler.ts line 10	5	Maximum simultaneous HTTP requests
DEEPWIKI_REQUEST_TIMEOUT	process.env.DEEPWIKI_REQUEST_TIMEOUT	30000	Request timeout in milliseconds
DEEPWIKI_MAX_RETRIES	RETRY_LIMIT constant in src/lib/httpCrawler.ts line 11	3	Maximum retry attempts for failed requests
DEEPWIKI_RETRY_DELAY	BACKOFF_BASE_MS constant in src/lib/httpCrawler.ts line 12	250	Base delay in milliseconds for exponential backoff

Note that DEEPWIKI_MAX_CONCURRENCY is accessed via process.env.DEEPWIKI_CONCURRENCY in the source code, while DEEPWIKI_MAX_RETRIES and DEEPWIKI_RETRY_DELAY map to internal constants RETRY_LIMIT and BACKOFF_BASE_MS respectively.

Configuration Implementation Details

The environment variables are consumed during crawler initialization in src/lib/httpCrawler.ts at lines 10-12. The README.md at line 222 documents the timeout variable for end-user reference.

Concurrency Control

In src/lib/httpCrawler.ts line 10, the code reads:

const MAX_CONCURRENCY = parseInt(process.env.DEEPWIKI_CONCURRENCY || '5', 10);

This value limits the number of simultaneous connections to prevent overwhelming target servers and avoid rate limiting.

Retry Logic with Exponential Backoff

Lines 11-12 in the same file define the resilience parameters:

const RETRY_LIMIT = parseInt(process.env.DEEPWIKI_MAX_RETRIES || '3', 10);
const BACKOFF_BASE_MS = parseInt(process.env.DEEPWIKI_RETRY_DELAY || '250', 10);

The crawler uses these values to implement exponential backoff, calculating delay as BACKOFF_BASE_MS * (2 ** attempt).

Practical Configuration Examples

You can set these variables via command line, environment files, or programmatically before initializing the crawler.

Command-Line Configuration

Export variables before running the MCP server:

export DEEPWIKI_MAX_CONCURRENCY=10
export DEEPWIKI_REQUEST_TIMEOUT=60000
export DEEPWIKI_MAX_RETRIES=5
export DEEPWIKI_RETRY_DELAY=500

npx mcp-deepwiki

Or inline for a single execution:

DEEPWIKI_MAX_CONCURRENCY=10 DEEPWIKI_REQUEST_TIMEOUT=60000 npx mcp-deepwiki

Environment File Setup

Create a .env file in your project root:

DEEPWIKI_MAX_CONCURRENCY=8
DEEPWIKI_REQUEST_TIMEOUT=45000
DEEPWIKI_MAX_RETRIES=4
DEEPWIKI_RETRY_DELAY=300

Load it with dotenv-cli or your preferred loader:

dotenv -e .env -- npx mcp-deepwiki

Programmatic Configuration

When using the crawler library directly, set environment variables before importing or invoking the crawl function:

process.env.DEEPWIKI_CONCURRENCY = '12';
process.env.DEEPWIKI_REQUEST_TIMEOUT = '50000';
process.env.DEEPWIKI_MAX_RETRIES = '6';
process.env.DEEPWIKI_RETRY_DELAY = '400';

import { crawl } from 'deepwiki-mcp';
await crawl({ root: 'https://example.com', maxDepth: 3 });

Note that when setting programmatically, use DEEPWIKI_CONCURRENCY rather than DEEPWIKI_MAX_CONCURRENCY to match the internal variable name used in src/lib/httpCrawler.ts.

Summary

• Four environment variables govern deepwiki-mcp's HTTP behavior: DEEPWIKI_MAX_CONCURRENCY, DEEPWIKI_REQUEST_TIMEOUT, DEEPWIKI_MAX_RETRIES, and DEEPWIKI_RETRY_DELAY.
• Default values provide conservative limits: 5 concurrent requests, 30-second timeout, 3 retries, and 250ms base delay.
• Source implementation resides in src/lib/httpCrawler.ts lines 10-12, where variables are parsed as integers with fallback defaults.
• Configuration methods include shell exports, .env files, and programmatic assignment before crawler initialization.

Frequently Asked Questions

What is the default request timeout in deepwiki-mcp?

The default request timeout is 30000 milliseconds (30 seconds). This value is documented in the README.md at line 222 and applies when DEEPWIKI_REQUEST_TIMEOUT is not explicitly set in your environment.

Why does the concurrency variable use a different name in the source code?

While the documentation refers to DEEPWIKI_MAX_CONCURRENCY, the source code in src/lib/httpCrawler.ts line 10 reads process.env.DEEPWIKI_CONCURRENCY. Both names refer to the same configuration, but the environment variable should be set as DEEPWIKI_CONCURRENCY when targeting the code directly, or DEEPWIKI_MAX_CONCURRENCY when following documentation conventions.

How does the retry delay work with exponential backoff?

The DEEPWIKI_RETRY_DELAY value (default 250ms) serves as the base for exponential backoff calculations. The actual delay between retries increases exponentially using the formula BACKOFF_BASE_MS * (2 ** attempt), meaning the first retry waits 250ms, the second waits 500ms, the third waits 1000ms, and so on until reaching the DEEPWIKI_MAX_RETRIES limit.

Can I disable retries or concurrency limits entirely?

While you cannot explicitly disable these features through environment variables, you can effectively neutralize them by setting extreme values. Set DEEPWIKI_MAX_RETRIES to 0 to prevent retry attempts, or set DEEPWIKI_MAX_CONCURRENCY to 1 to force sequential processing. However, setting concurrency too high or retries to zero may cause rate limiting or complete failure on transient network errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}