How httpCrawler Implements Breadth-First Traversal in Deepwiki-MCP

The httpCrawler module in deepwiki-mcp implements breadth-first traversal using a FIFO-based PQueue that processes URLs level-by-level, enqueuing child links only after their parent page is fetched while respecting configurable depth limits and domain restrictions.

The deepwiki-mcp repository provides a robust HTTP crawling solution designed for systematic web documentation extraction. At its core, the httpCrawler module implements a breadth-first traversal algorithm that ensures comprehensive site coverage while maintaining strict crawling policies and resilience mechanisms.

Core Breadth-First Implementation in httpCrawler.ts

The breadth-first traversal logic resides primarily in [src/lib/httpCrawler.ts](https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts), where the algorithm combines a FIFO task queue with depth-tracking to achieve level-by-level crawling.

FIFO Queue Architecture with PQueue

The crawler utilizes a PQueue instance configured with concurrency: MAX_CONCURRENCY to manage the crawl frontier. As noted in the source code at line 34, the queue processes tasks in first-in-first-out order. When the enqueue function discovers child URLs, it adds them to the queue using queue.add(async () => { … }), ensuring that all URLs at depth d are processed before any URLs at depth d + 1.

Depth-Limited Traversal Logic

The internal async function enqueue(url, depth) implements the core BFS logic referenced in the comment at line 28. The function validates the URL against file-type filters, depth limits, and domain restrictions before queuing. Crucially, child URLs are enqueued with await enqueue(child, depth + 1) only after their parent page is successfully retrieved, guaranteeing that the crawl proceeds level by level rather than depth-first.

Crawling Constraints and Policies

Beyond the basic breadth-first traversal algorithm, the crawler enforces several policies to ensure respectful and targeted crawling.

Same-Origin Enforcement

The crawler restricts traversal to the initial hostname using the check if (url.hostname !== root.hostname) return. This same-origin policy prevents the crawler from wandering into external domains while maintaining the breadth-first structure within the target site.

Robots.txt Compliance

Before crawling begins, the system fetches and parses robots.txt using robotsParser. Each URL is vetted with robots.isAllowed(url.href, '*') before enqueueing, ensuring the breadth-first traversal respects site-wide crawling policies.

Resilience and Concurrency Controls

The implementation includes robust error handling and concurrency management to maintain reliable breadth-first traversal under network volatility.

Exponential Backoff Retry Mechanism

Failed requests are retried up to RETRY_LIMIT times with exponential backoff calculated as BACKOFF_BASE_MS * 2 ** (retries - 1). This ensures temporary network failures do not break the breadth-first traversal flow while preventing aggressive re-requests.

Configurable Concurrency Limits

The crawler reads the DEEPWIKI_CONCURRENCY environment variable at runtime (defaulting to 5) to set MAX_CONCURRENCY. This controls how many URLs from the BFS queue are processed simultaneously, balancing speed against server load.

Practical Usage Examples

Basic Breadth-First Crawl

import { crawl } from '@/lib/httpCrawler'
import { URL } from 'node:url'

const root = new URL('https://example.com/')
const maxDepth = 3

await crawl({
  root,
  maxDepth,
  emit: (e) => {
    console.log(`[${e.type}] ${e.url} – ${e.bytes} B fetched (${e.retries} retries)`)
  },
  verbose: true,
})

This configuration initiates a breadth-first crawl starting at example.com, processing all depth-1 pages before depth-2 pages, up to the specified maximum depth.

Adjusting Concurrency and Retry Behavior

process.env.DEEPWIKI_CONCURRENCY = '10'

await crawl({
  root: new URL('https://mywiki.org/'),
  maxDepth: 2,
  emit: e => console.log(e),
})

Setting the environment variable before invocation increases the BFS concurrency, allowing up to 10 simultaneous requests while maintaining the level-by-level traversal order.

Handling Crawl Results

const result = await crawl({ root, maxDepth, emit: () => {} })

console.log('Fetched pages:', Object.keys(result.html).length)
console.log('Total bytes:', result.bytes)
console.log('Errors:', result.errors)

The returned object contains the complete breadth-first crawl results, including HTML content mapped by path and any URLs that failed after retry limits were exhausted.

Summary

• The httpCrawler implements breadth-first traversal using a FIFO PQueue that processes URLs level-by-level as defined in src/lib/httpCrawler.ts.
• Child URLs are enqueued with incremented depth only after their parent page is successfully fetched, ensuring strict BFS ordering.
• The algorithm respects same-origin policies, robots.txt directives, and configurable depth limits to ensure targeted, respectful crawling.
• Exponential backoff and configurable concurrency (via DEEPWIKI_CONCURRENCY) provide resilience while maintaining the breadth-first structure.

Frequently Asked Questions

How does httpCrawler maintain breadth-first order instead of depth-first?

The crawler uses a PQueue instance configured as a FIFO (first-in-first-out) queue. When the enqueue function discovers links on a page, it adds them to the queue using queue.add(). Since the queue processes tasks in the order they were added, and children are only enqueued after their parent completes, all URLs at depth d are processed before any URLs at depth d + 1, ensuring strict breadth-first traversal.

What limits the depth of the breadth-first traversal?

The maxDepth parameter passed to the crawl function restricts how many levels the BFS algorithm explores. Inside the enqueue function, the crawler checks the current depth against this limit before adding URLs to the queue. Additionally, the depth variable is incremented only when enqueuing children (enqueue(child, depth + 1)), ensuring the level boundary is respected throughout the breadth-first crawl.

How does the crawler handle concurrent requests while maintaining BFS ordering?

The crawler uses PQueue with a configurable concurrency limit (set via the DEEPWIKI_CONCURRENCY environment variable, defaulting to 5). While multiple requests run simultaneously, the FIFO nature of the queue ensures that the order in which URLs are added determines the processing sequence. This means the breadth-first structure is maintained at the logical level (which URLs are discovered and queued), even while the physical HTTP requests execute concurrently for performance.

Does the breadth-first traversal respect robots.txt restrictions?

Yes, before any URL is added to the BFS queue, the crawler validates it against the site's robots.txt file. The crawler fetches and parses robots.txt once at startup using robotsParser, then checks each candidate URL with robots.isAllowed(url.href, '*') inside the enqueue function. Only URLs permitted by robots.txt are added to the breadth-first queue, ensuring the traversal respects site crawling policies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}