# How httpCrawler Implements Breadth-First Traversal in Deepwiki-MCP

> Discover how httpCrawler uses FIFO PQueue for level-by-level breadth-first traversal in deepwiki-mcp respecting depth and domain limits. Optimize your deep web crawling.

- Repository: [Kevin Kern/deepwiki-mcp](https://github.com/regenrek/deepwiki-mcp)
- Tags: internals
- Published: 2026-02-16

---

**The httpCrawler module in deepwiki-mcp implements breadth-first traversal using a FIFO-based PQueue that processes URLs level-by-level, enqueuing child links only after their parent page is fetched while respecting configurable depth limits and domain restrictions.**

The deepwiki-mcp repository provides a robust HTTP crawling solution designed for systematic web documentation extraction. At its core, the `httpCrawler` module implements a **breadth-first traversal** algorithm that ensures comprehensive site coverage while maintaining strict crawling policies and resilience mechanisms.

## Core Breadth-First Implementation in httpCrawler.ts

The breadth-first traversal logic resides primarily in [[`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts)](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts), where the algorithm combines a FIFO task queue with depth-tracking to achieve level-by-level crawling.

### FIFO Queue Architecture with PQueue

The crawler utilizes a `PQueue` instance configured with `concurrency: MAX_CONCURRENCY` to manage the crawl frontier. As noted in the source code at line 34, the queue processes tasks in **first-in-first-out** order. When the `enqueue` function discovers child URLs, it adds them to the queue using `queue.add(async () => { … })`, ensuring that all URLs at depth *d* are processed before any URLs at depth *d + 1*.

### Depth-Limited Traversal Logic

The internal `async function enqueue(url, depth)` implements the core BFS logic referenced in the comment at line 28. The function validates the URL against file-type filters, depth limits, and domain restrictions before queuing. Crucially, child URLs are enqueued with `await enqueue(child, depth + 1)` only after their parent page is successfully retrieved, guaranteeing that the crawl proceeds level by level rather than depth-first.

## Crawling Constraints and Policies

Beyond the basic breadth-first traversal algorithm, the crawler enforces several policies to ensure respectful and targeted crawling.

### Same-Origin Enforcement

The crawler restricts traversal to the initial hostname using the check `if (url.hostname !== root.hostname) return`. This same-origin policy prevents the crawler from wandering into external domains while maintaining the breadth-first structure within the target site.

### Robots.txt Compliance

Before crawling begins, the system fetches and parses [`robots.txt`](https://github.com/regenrek/deepwiki-mcp/blob/main/robots.txt) using `robotsParser`. Each URL is vetted with `robots.isAllowed(url.href, '*')` before enqueueing, ensuring the breadth-first traversal respects site-wide crawling policies.

## Resilience and Concurrency Controls

The implementation includes robust error handling and concurrency management to maintain reliable breadth-first traversal under network volatility.

### Exponential Backoff Retry Mechanism

Failed requests are retried up to `RETRY_LIMIT` times with exponential backoff calculated as `BACKOFF_BASE_MS * 2 ** (retries - 1)`. This ensures temporary network failures do not break the breadth-first traversal flow while preventing aggressive re-requests.

### Configurable Concurrency Limits

The crawler reads the `DEEPWIKI_CONCURRENCY` environment variable at runtime (defaulting to 5) to set `MAX_CONCURRENCY`. This controls how many URLs from the BFS queue are processed simultaneously, balancing speed against server load.

## Practical Usage Examples

### Basic Breadth-First Crawl

```typescript
import { crawl } from '@/lib/httpCrawler'
import { URL } from 'node:url'

const root = new URL('https://example.com/')
const maxDepth = 3

await crawl({
  root,
  maxDepth,
  emit: (e) => {
    console.log(`[${e.type}] ${e.url} – ${e.bytes} B fetched (${e.retries} retries)`)
  },
  verbose: true,
})

```

This configuration initiates a breadth-first crawl starting at `example.com`, processing all depth-1 pages before depth-2 pages, up to the specified maximum depth.

### Adjusting Concurrency and Retry Behavior

```typescript
process.env.DEEPWIKI_CONCURRENCY = '10'

await crawl({
  root: new URL('https://mywiki.org/'),
  maxDepth: 2,
  emit: e => console.log(e),
})

```

Setting the environment variable before invocation increases the BFS concurrency, allowing up to 10 simultaneous requests while maintaining the level-by-level traversal order.

### Handling Crawl Results

```typescript
const result = await crawl({ root, maxDepth, emit: () => {} })

console.log('Fetched pages:', Object.keys(result.html).length)
console.log('Total bytes:', result.bytes)
console.log('Errors:', result.errors)

```

The returned object contains the complete breadth-first crawl results, including HTML content mapped by path and any URLs that failed after retry limits were exhausted.

## Summary

- The **httpCrawler** implements breadth-first traversal using a **FIFO PQueue** that processes URLs level-by-level as defined in [`src/lib/httpCrawler.ts`](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts).
- Child URLs are enqueued with incremented depth only after their parent page is successfully fetched, ensuring strict BFS ordering.
- The algorithm respects **same-origin policies**, **robots.txt** directives, and configurable **depth limits** to ensure targeted, respectful crawling.
- **Exponential backoff** and **configurable concurrency** (via `DEEPWIKI_CONCURRENCY`) provide resilience while maintaining the breadth-first structure.

## Frequently Asked Questions

### How does httpCrawler maintain breadth-first order instead of depth-first?

The crawler uses a `PQueue` instance configured as a FIFO (first-in-first-out) queue. When the `enqueue` function discovers links on a page, it adds them to the queue using `queue.add()`. Since the queue processes tasks in the order they were added, and children are only enqueued after their parent completes, all URLs at depth *d* are processed before any URLs at depth *d + 1*, ensuring strict breadth-first traversal.

### What limits the depth of the breadth-first traversal?

The `maxDepth` parameter passed to the `crawl` function restricts how many levels the BFS algorithm explores. Inside the `enqueue` function, the crawler checks the current depth against this limit before adding URLs to the queue. Additionally, the `depth` variable is incremented only when enqueuing children (`enqueue(child, depth + 1)`), ensuring the level boundary is respected throughout the breadth-first crawl.

### How does the crawler handle concurrent requests while maintaining BFS ordering?

The crawler uses `PQueue` with a configurable `concurrency` limit (set via the `DEEPWIKI_CONCURRENCY` environment variable, defaulting to 5). While multiple requests run simultaneously, the FIFO nature of the queue ensures that the order in which URLs are added determines the processing sequence. This means the breadth-first structure is maintained at the logical level (which URLs are discovered and queued), even while the physical HTTP requests execute concurrently for performance.

### Does the breadth-first traversal respect robots.txt restrictions?

Yes, before any URL is added to the BFS queue, the crawler validates it against the site's robots.txt file. The crawler fetches and parses robots.txt once at startup using `robotsParser`, then checks each candidate URL with `robots.isAllowed(url.href, '*')` inside the `enqueue` function. Only URLs permitted by robots.txt are added to the breadth-first queue, ensuring the traversal respects site crawling policies.