How OpenSEO Discovers URLs from robots.txt and Sitemaps: A Technical Deep Dive

OpenSEO discovers crawlable URLs by fetching and parsing robots.txt to extract sitemap locations, recursively fetching those sitemaps (including nested indexes) up to a depth of 3, and deduplicating the results while respecting crawl budgets and origin boundaries.

OpenSEO is an open-source site audit tool that implements a deterministic URL discovery pipeline to build seed lists for SEO analysis. The system intelligently combines robots.txt parsing with recursive sitemap index traversal to find crawlable pages while respecting site boundaries and resource limits. This article examines the implementation details found in src/server/lib/audit/discovery.ts and supporting utilities.

Fetching and Parsing robots.txt

The discovery process begins with retrieving the site's robots.txt file to identify sitemap declarations and crawl permissions.

Retrieving the robots.txt File

The fetchRobotsTxtText function in src/server/lib/audit/discovery.ts issues a GET request to <origin>/robots.txt with a short-lived User-Agent header and a 10-second timeout. If successful, the response body is trimmed to 500 KB (the RFC-recommended cap) before being returned; otherwise, the function returns null.

// src/server/lib/audit/discovery.ts#L40-L51
const txt = await fetchRobotsTxtText("https://example.com");
console.log(txt); // Raw robots.txt content or null

Parsing with robots-parser Library

The parseRobotsTxt function passes the raw text (or null) to the robots-parser library. When the file is missing, it yields a permissive result where isAllowed always returns true and the sitemap list is empty. When present, it returns a callable isAllowed function and an array of sitemap URLs declared via Sitemap: directives.

// src/server/lib/audit/discovery.ts#L55-L69
const { isAllowed, sitemapUrls } = parseRobotsTxt("https://example.com", txt);
console.log(sitemapUrls); // Array of sitemap URLs from robots.txt

Sitemap Discovery and Collection

OpenSEO aggregates sitemap locations from multiple sources to ensure comprehensive coverage.

Conventional Fallbacks and Sitemap Sources

The discoverUrls function builds a Set containing every sitemap URL returned by the parser plus the conventional fallback location <origin>/sitemap.xml. This dual-source approach ensures that sites with undeclared but standard sitemap locations are still processed correctly.

// src/server/lib/audit/discovery.ts#L33-L36
// Combines parsed sitemaps with conventional fallback
const sitemapSources = new Set([
  ...parsedSitemaps,
  new URL("/sitemap.xml", origin).href
]);

Recursive Sitemap Processing

Once sitemap sources are identified, OpenSEO recursively fetches and parses the XML documents to extract page URLs.

Fetching and Validating Sitemap Documents

The fetchSitemapDocumentWithRetry function retrieves sitemap URLs with a 15-second timeout, verifies the response remains on the same origin using isSameOrigin from src/server/lib/audit/url-utils.ts, and reads the body up to 10 MB via readBodyCapped. This prevents resource exhaustion from oversized sitemap files.

// Fetches with retry logic and origin validation
const doc = await fetchSitemapDocumentWithRetry(sitemapUrl, origin);

Extracting URLs from XML Sitemaps

The XML is parsed using fast-xml-parser. The helper getParsedSitemapSections splits the result into sitemap (nested indexes) and url (page entries) sections. The getSitemapLocations function extracts <loc> values, normalizes them with normalizeUrl, and filters invalid URLs, returning two arrays: nestedSitemaps (further indexes to crawl) and pageUrls (individual pages ready for audit).

const sections = getParsedSitemapSections(parsedXml);
const { nestedSitemaps, pageUrls } = getSitemapLocations(sections);

Breadth-First Traversal Algorithm

OpenSEO implements a breadth-first search with strict limits to prevent runaway crawling of deeply nested sitemap indexes.

Depth Limits and Deduplication

The discoverUrls function maintains a queue of {url, depth} objects. For each entry, it checks origin boundaries, verifies against seenSitemapDocs (a deduplication set), and processes the document. Discovered nested sitemaps are enqueued with a decremented depth, defaulting to a maximum depth of MAX_SITEMAP_DEPTH = 3.

Budget Constraints and Safety Limits

The traversal loop terminates when:

• The queue empties (all sitemaps processed)
• MAX_SITEMAP_DOCS = 300 unique sitemap documents are fetched
• The maxPages URL budget is satisfied (defaulting to 50 URLs)

Page URLs are added to the master allUrls set until the budget is reached, ensuring the crawler remains resource-efficient.

// src/server/lib/audit/discovery.ts#L82-L99
// Queue processing with depth tracking and budget enforcement
while (queue.length > 0 && allUrls.size < maxPages) {
  const { url, depth } = queue.shift();
  if (depth <= 0 || seenSitemapDocs.has(url)) continue;
  // Process document, extract URLs, enqueue nested sitemaps with depth - 1
}

Implementation Example

You can invoke the discovery pipeline directly or use the low-level helpers for custom implementations:

// Complete pipeline: Get up to 30 crawl-seed URLs
import { discoverUrls } from "@/server/lib/audit/discovery";

const { urls, robotsText } = await discoverUrls("https://example.com", 30);
console.log(urls);        // Array of up to 30 page URLs
console.log(robotsText);  // Raw robots.txt content or null

// Manual use of low-level helpers
import { fetchRobotsTxtText, parseRobotsTxt } from "@/server/lib/audit/discovery";

const txt = await fetchRobotsTxtText("https://example.com");
const { sitemapUrls } = parseRobotsTxt("https://example.com", txt);
console.log(sitemapUrls); // Sitemaps declared in robots.txt

Summary

• OpenSEO fetches robots.txt with a 10-second timeout and 500 KB size cap, parsing it with robots-parser to extract sitemap URLs and permission rules.
• The system adds a conventional /sitemap.xml fallback to ensure comprehensive coverage even when robots.txt lacks sitemap declarations.
• Sitemap documents are fetched with a 15-second timeout, 10 MB size limit, and strict same-origin validation via isSameOrigin in url-utils.ts.
• fast-xml-parser processes the XML, separating nested sitemap indexes from actual page URLs while normalizing all discovered locations.
• A breadth-first queue processes sitemaps up to MAX_SITEMAP_DEPTH = 3 and MAX_SITEMAP_DOCS = 300, deduplicating entries and respecting the maxPages budget.
• The final output includes a capped array of unique, same-origin page URLs and the raw robots.txt content for downstream analysis in page-analyzer.ts.

Frequently Asked Questions

What happens if a site has no robots.txt file?

If fetchRobotsTxtText returns null, the parseRobotsTxt function yields a permissive configuration where isAllowed always returns true and the sitemap list is empty. The discovery process then relies solely on the conventional /sitemap.xml fallback and any manually provided sitemap sources.

How does OpenSEO handle nested sitemap indexes?

The system uses getParsedSitemapSections to distinguish between <sitemap> (index entries) and <url> (page entries) elements. Nested sitemaps are enqueued with a decremented depth counter, allowing recursive processing up to the default limit of 3 levels deep while preventing infinite recursion through the seenSitemapDocs deduplication set.

What safety limits prevent the crawler from consuming excessive resources?

OpenSEO implements multiple hard limits: 500 KB for robots.txt responses, 10 MB for individual sitemap files, 10-second and 15-second timeouts for fetch operations, maximum 300 unique sitemap documents, and a default 50 URL output cap. These constraints ensure the discovery phase completes quickly without overwhelming the target server or the auditing infrastructure.

Can the discovery process be customized for different crawl budgets?

Yes. The discoverUrls function accepts a maxPages parameter (default 50) to control the output size, and the internal constants MAX_SITEMAP_DEPTH and MAX_SITEMAP_DOCS can be modified in the source to adjust traversal depth and document limits according to specific auditing requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}