# How OpenSEO Discovers URLs from robots.txt and Sitemaps: A Technical Deep Dive

> Learn how OpenSEO finds crawlable URLs by parsing robots.txt and sitemaps. Discover the technical details of URL discovery and sitemap indexing up to 3 levels deep.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**OpenSEO discovers crawlable URLs by fetching and parsing robots.txt to extract sitemap locations, recursively fetching those sitemaps (including nested indexes) up to a depth of 3, and deduplicating the results while respecting crawl budgets and origin boundaries.**

OpenSEO is an open-source site audit tool that implements a deterministic URL discovery pipeline to build seed lists for SEO analysis. The system intelligently combines **robots.txt** parsing with recursive **sitemap index** traversal to find crawlable pages while respecting site boundaries and resource limits. This article examines the implementation details found in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) and supporting utilities.

## Fetching and Parsing robots.txt

The discovery process begins with retrieving the site's robots.txt file to identify sitemap declarations and crawl permissions.

### Retrieving the robots.txt File

The `fetchRobotsTxtText` function in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) issues a GET request to `<origin>/robots.txt` with a short-lived User-Agent header and a **10-second timeout**. If successful, the response body is trimmed to **500 KB** (the RFC-recommended cap) before being returned; otherwise, the function returns `null`.

```typescript
// src/server/lib/audit/discovery.ts#L40-L51
const txt = await fetchRobotsTxtText("https://example.com");
console.log(txt); // Raw robots.txt content or null

```

### Parsing with robots-parser Library

The `parseRobotsTxt` function passes the raw text (or `null`) to the `robots-parser` library. When the file is missing, it yields a permissive result where `isAllowed` always returns `true` and the sitemap list is empty. When present, it returns a callable `isAllowed` function and an array of sitemap URLs declared via `Sitemap:` directives.

```typescript
// src/server/lib/audit/discovery.ts#L55-L69
const { isAllowed, sitemapUrls } = parseRobotsTxt("https://example.com", txt);
console.log(sitemapUrls); // Array of sitemap URLs from robots.txt

```

## Sitemap Discovery and Collection

OpenSEO aggregates sitemap locations from multiple sources to ensure comprehensive coverage.

### Conventional Fallbacks and Sitemap Sources

The `discoverUrls` function builds a `Set` containing every sitemap URL returned by the parser plus the conventional fallback location `<origin>/sitemap.xml`. This dual-source approach ensures that sites with undeclared but standard sitemap locations are still processed correctly.

```typescript
// src/server/lib/audit/discovery.ts#L33-L36
// Combines parsed sitemaps with conventional fallback
const sitemapSources = new Set([
  ...parsedSitemaps,
  new URL("/sitemap.xml", origin).href
]);

```

## Recursive Sitemap Processing

Once sitemap sources are identified, OpenSEO recursively fetches and parses the XML documents to extract page URLs.

### Fetching and Validating Sitemap Documents

The `fetchSitemapDocumentWithRetry` function retrieves sitemap URLs with a **15-second timeout**, verifies the response remains on the same origin using `isSameOrigin` from [`src/server/lib/audit/url-utils.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-utils.ts), and reads the body up to **10 MB** via `readBodyCapped`. This prevents resource exhaustion from oversized sitemap files.

```typescript
// Fetches with retry logic and origin validation
const doc = await fetchSitemapDocumentWithRetry(sitemapUrl, origin);

```

### Extracting URLs from XML Sitemaps

The XML is parsed using `fast-xml-parser`. The helper `getParsedSitemapSections` splits the result into **sitemap** (nested indexes) and **url** (page entries) sections. The `getSitemapLocations` function extracts `<loc>` values, normalizes them with `normalizeUrl`, and filters invalid URLs, returning two arrays: `nestedSitemaps` (further indexes to crawl) and `pageUrls` (individual pages ready for audit).

```typescript
const sections = getParsedSitemapSections(parsedXml);
const { nestedSitemaps, pageUrls } = getSitemapLocations(sections);

```

## Breadth-First Traversal Algorithm

OpenSEO implements a **breadth-first search** with strict limits to prevent runaway crawling of deeply nested sitemap indexes.

### Depth Limits and Deduplication

The `discoverUrls` function maintains a queue of `{url, depth}` objects. For each entry, it checks origin boundaries, verifies against `seenSitemapDocs` (a deduplication set), and processes the document. Discovered nested sitemaps are enqueued with a decremented depth, defaulting to a maximum depth of `MAX_SITEMAP_DEPTH = 3`.

### Budget Constraints and Safety Limits

The traversal loop terminates when:
- The queue empties (all sitemaps processed)
- `MAX_SITEMAP_DOCS = 300` unique sitemap documents are fetched
- The `maxPages` URL budget is satisfied (defaulting to 50 URLs)

Page URLs are added to the master `allUrls` set until the budget is reached, ensuring the crawler remains resource-efficient.

```typescript
// src/server/lib/audit/discovery.ts#L82-L99
// Queue processing with depth tracking and budget enforcement
while (queue.length > 0 && allUrls.size < maxPages) {
  const { url, depth } = queue.shift();
  if (depth <= 0 || seenSitemapDocs.has(url)) continue;
  // Process document, extract URLs, enqueue nested sitemaps with depth - 1
}

```

## Implementation Example

You can invoke the discovery pipeline directly or use the low-level helpers for custom implementations:

```typescript
// Complete pipeline: Get up to 30 crawl-seed URLs
import { discoverUrls } from "@/server/lib/audit/discovery";

const { urls, robotsText } = await discoverUrls("https://example.com", 30);
console.log(urls);        // Array of up to 30 page URLs
console.log(robotsText);  // Raw robots.txt content or null

```

```typescript
// Manual use of low-level helpers
import { fetchRobotsTxtText, parseRobotsTxt } from "@/server/lib/audit/discovery";

const txt = await fetchRobotsTxtText("https://example.com");
const { sitemapUrls } = parseRobotsTxt("https://example.com", txt);
console.log(sitemapUrls); // Sitemaps declared in robots.txt

```

## Summary

- **OpenSEO** fetches robots.txt with a 10-second timeout and 500 KB size cap, parsing it with `robots-parser` to extract sitemap URLs and permission rules.
- The system adds a conventional [`/sitemap.xml`](https://github.com/every-app/open-seo/blob/main//sitemap.xml) fallback to ensure comprehensive coverage even when robots.txt lacks sitemap declarations.
- Sitemap documents are fetched with a 15-second timeout, 10 MB size limit, and strict same-origin validation via `isSameOrigin` in [`url-utils.ts`](https://github.com/every-app/open-seo/blob/main/url-utils.ts).
- **fast-xml-parser** processes the XML, separating nested sitemap indexes from actual page URLs while normalizing all discovered locations.
- A breadth-first queue processes sitemaps up to `MAX_SITEMAP_DEPTH = 3` and `MAX_SITEMAP_DOCS = 300`, deduplicating entries and respecting the `maxPages` budget.
- The final output includes a capped array of unique, same-origin page URLs and the raw robots.txt content for downstream analysis in [`page-analyzer.ts`](https://github.com/every-app/open-seo/blob/main/page-analyzer.ts).

## Frequently Asked Questions

### What happens if a site has no robots.txt file?

If `fetchRobotsTxtText` returns `null`, the `parseRobotsTxt` function yields a permissive configuration where `isAllowed` always returns `true` and the sitemap list is empty. The discovery process then relies solely on the conventional [`/sitemap.xml`](https://github.com/every-app/open-seo/blob/main//sitemap.xml) fallback and any manually provided sitemap sources.

### How does OpenSEO handle nested sitemap indexes?

The system uses `getParsedSitemapSections` to distinguish between `<sitemap>` (index entries) and `<url>` (page entries) elements. Nested sitemaps are enqueued with a decremented depth counter, allowing recursive processing up to the default limit of 3 levels deep while preventing infinite recursion through the `seenSitemapDocs` deduplication set.

### What safety limits prevent the crawler from consuming excessive resources?

OpenSEO implements multiple hard limits: **500 KB** for robots.txt responses, **10 MB** for individual sitemap files, **10-second and 15-second timeouts** for fetch operations, maximum **300 unique sitemap documents**, and a default **50 URL** output cap. These constraints ensure the discovery phase completes quickly without overwhelming the target server or the auditing infrastructure.

### Can the discovery process be customized for different crawl budgets?

Yes. The `discoverUrls` function accepts a `maxPages` parameter (default 50) to control the output size, and the internal constants `MAX_SITEMAP_DEPTH` and `MAX_SITEMAP_DOCS` can be modified in the source to adjust traversal depth and document limits according to specific auditing requirements.