# Environment Variables That Control Concurrency, Timeout, and Retry Behavior in deepwiki-mcp

> Master deepwiki-mcp concurrency, timeout, and retry behavior using environment variables like DEEPWIKI_MAX_CONCURRENCY and DEEPWIKI_REQUEST_TIMEOUT. Optimize your crawler performance now.

- Repository: [Kevin Kern/deepwiki-mcp](https://github.com/regenrek/deepwiki-mcp)
- Tags: how-to-guide
- Published: 2026-02-16

---

**The deepwiki-mcp crawler respects four environment variables—`DEEPWIKI_MAX_CONCURRENCY`, `DEEPWIKI_REQUEST_TIMEOUT`, `DEEPWIKI_MAX_RETRIES`, and `DEEPWIKI_RETRY_DELAY`—which default to 5 concurrent requests, 30000ms timeout, 3 retry attempts, and 250ms base delay respectively.**

The `regenrek/deepwiki-mcp` repository provides a Model Context Protocol (MCP) server for crawling and analyzing documentation sources. Understanding which environment variables control concurrency, timeout, and retry behavior in deepwiki-mcp enables precise tuning of HTTP request handling to prevent rate limiting and gracefully handle transient network failures.

## Core Environment Variables for HTTP Crawling

The crawler configuration relies on four optional environment variables defined in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts). When omitted, the system applies conservative defaults suitable for most documentation crawling scenarios.

| Variable | Source Code Reference | Default | Purpose |
|----------|----------------------|---------|---------|
| `DEEPWIKI_MAX_CONCURRENCY` | `process.env.DEEPWIKI_CONCURRENCY` in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) line 10 | `5` | Maximum simultaneous HTTP requests |
| `DEEPWIKI_REQUEST_TIMEOUT` | `process.env.DEEPWIKI_REQUEST_TIMEOUT` | `30000` | Request timeout in milliseconds |
| `DEEPWIKI_MAX_RETRIES` | `RETRY_LIMIT` constant in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) line 11 | `3` | Maximum retry attempts for failed requests |
| `DEEPWIKI_RETRY_DELAY` | `BACKOFF_BASE_MS` constant in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) line 12 | `250` | Base delay in milliseconds for exponential backoff |

Note that `DEEPWIKI_MAX_CONCURRENCY` is accessed via `process.env.DEEPWIKI_CONCURRENCY` in the source code, while `DEEPWIKI_MAX_RETRIES` and `DEEPWIKI_RETRY_DELAY` map to internal constants `RETRY_LIMIT` and `BACKOFF_BASE_MS` respectively.

## Configuration Implementation Details

The environment variables are consumed during crawler initialization in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) at lines 10-12. The README.md at line 222 documents the timeout variable for end-user reference.

### Concurrency Control

In [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) line 10, the code reads:

```typescript
const MAX_CONCURRENCY = parseInt(process.env.DEEPWIKI_CONCURRENCY || '5', 10);

```

This value limits the number of simultaneous connections to prevent overwhelming target servers and avoid rate limiting.

### Retry Logic with Exponential Backoff

Lines 11-12 in the same file define the resilience parameters:

```typescript
const RETRY_LIMIT = parseInt(process.env.DEEPWIKI_MAX_RETRIES || '3', 10);
const BACKOFF_BASE_MS = parseInt(process.env.DEEPWIKI_RETRY_DELAY || '250', 10);

```

The crawler uses these values to implement exponential backoff, calculating delay as `BACKOFF_BASE_MS * (2 ** attempt)`.

## Practical Configuration Examples

You can set these variables via command line, environment files, or programmatically before initializing the crawler.

### Command-Line Configuration

Export variables before running the MCP server:

```bash
export DEEPWIKI_MAX_CONCURRENCY=10
export DEEPWIKI_REQUEST_TIMEOUT=60000
export DEEPWIKI_MAX_RETRIES=5
export DEEPWIKI_RETRY_DELAY=500

npx mcp-deepwiki

```

Or inline for a single execution:

```bash
DEEPWIKI_MAX_CONCURRENCY=10 DEEPWIKI_REQUEST_TIMEOUT=60000 npx mcp-deepwiki

```

### Environment File Setup

Create a `.env` file in your project root:

```dotenv
DEEPWIKI_MAX_CONCURRENCY=8
DEEPWIKI_REQUEST_TIMEOUT=45000
DEEPWIKI_MAX_RETRIES=4
DEEPWIKI_RETRY_DELAY=300

```

Load it with `dotenv-cli` or your preferred loader:

```bash
dotenv -e .env -- npx mcp-deepwiki

```

### Programmatic Configuration

When using the crawler library directly, set environment variables before importing or invoking the crawl function:

```typescript
process.env.DEEPWIKI_CONCURRENCY = '12';
process.env.DEEPWIKI_REQUEST_TIMEOUT = '50000';
process.env.DEEPWIKI_MAX_RETRIES = '6';
process.env.DEEPWIKI_RETRY_DELAY = '400';

import { crawl } from 'deepwiki-mcp';
await crawl({ root: 'https://example.com', maxDepth: 3 });

```

Note that when setting programmatically, use `DEEPWIKI_CONCURRENCY` rather than `DEEPWIKI_MAX_CONCURRENCY` to match the internal variable name used in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts).

## Summary

- **Four environment variables** govern deepwiki-mcp's HTTP behavior: `DEEPWIKI_MAX_CONCURRENCY`, `DEEPWIKI_REQUEST_TIMEOUT`, `DEEPWIKI_MAX_RETRIES`, and `DEEPWIKI_RETRY_DELAY`.
- **Default values** provide conservative limits: 5 concurrent requests, 30-second timeout, 3 retries, and 250ms base delay.
- **Source implementation** resides in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) lines 10-12, where variables are parsed as integers with fallback defaults.
- **Configuration methods** include shell exports, `.env` files, and programmatic assignment before crawler initialization.

## Frequently Asked Questions

### What is the default request timeout in deepwiki-mcp?

The default request timeout is **30000 milliseconds (30 seconds)**. This value is documented in the README.md at line 222 and applies when `DEEPWIKI_REQUEST_TIMEOUT` is not explicitly set in your environment.

### Why does the concurrency variable use a different name in the source code?

While the documentation refers to `DEEPWIKI_MAX_CONCURRENCY`, the source code in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts) line 10 reads `process.env.DEEPWIKI_CONCURRENCY`. Both names refer to the same configuration, but the environment variable should be set as `DEEPWIKI_CONCURRENCY` when targeting the code directly, or `DEEPWIKI_MAX_CONCURRENCY` when following documentation conventions.

### How does the retry delay work with exponential backoff?

The `DEEPWIKI_RETRY_DELAY` value (default 250ms) serves as the base for exponential backoff calculations. The actual delay between retries increases exponentially using the formula `BACKOFF_BASE_MS * (2 ** attempt)`, meaning the first retry waits 250ms, the second waits 500ms, the third waits 1000ms, and so on until reaching the `DEEPWIKI_MAX_RETRIES` limit.

### Can I disable retries or concurrency limits entirely?

While you cannot explicitly disable these features through environment variables, you can effectively neutralize them by setting extreme values. Set `DEEPWIKI_MAX_RETRIES` to `0` to prevent retry attempts, or set `DEEPWIKI_MAX_CONCURRENCY` to `1` to force sequential processing. However, setting concurrency too high or retries to zero may cause rate limiting or complete failure on transient network errors.