How the Open-SEO Audit System Discovers URLs and Enforces robots.txt Policies

The Open-SEO audit system discovers URLs by recursively fetching sitemaps declared in robots.txt and enforces policies by validating every candidate link against parsed robots rules before queuing it for crawling.

The every-app/open-seo repository implements a deterministic two-phase audit workflow that completely separates URL discovery from the actual crawling process. This architecture ensures that robots.txt policies are fetched and parsed before any crawling begins, then applied consistently to every URL candidate during the crawl phase.

How URL Discovery Works

The discovery phase runs once at the start of each audit via runDiscoveryPhase, gathering all candidate URLs before any network requests to actual pages occur.

Fetching and Parsing robots.txt

The process begins in src/server/lib/audit/discovery.ts where the discoverUrls function orchestrates the initial fetch. At lines 29-38, fetchRobotsTxtText(origin) performs a GET request to <origin>/robots.txt with a 10-second timeout. If the request fails or returns a non-OK status, the function returns null rather than throwing.

At lines 44-58, parseRobotsTxt(origin, robotsText) processes the fetched text using the robots-parser library. When the input is null, it returns a permissive object where isAllowed always returns true and the sitemap list is empty. This ensures the audit can continue even when robots.txt is unreachable.

Recursive Sitemap Processing

The system extracts <sitemap> locations from the parsed robots.txt and stores them in robots.sitemapUrls (lines 91-94). It also automatically appends the conventional origin + '/sitemap.xml' as a fallback location.

The fetchSitemapDocumentWithRetry function (lines 61-78) handles the actual fetching with a 15-second timeout. It validates that responses contain XML, then parses them using fast-xml-parser to extract both nested sitemap references and actual page URLs. The recursion respects MAX_SITEMAP_DEPTH (set to 3) and MAX_SITEMAP_DOCS (set to 300), preventing infinite loops on circular references.

URL Collection and Limits

All discovered page URLs are deduplicated using a JavaScript Set named allUrls. The system calculates a maxDiscoveredUrls limit between 500 and 50,000 based on the maxPages parameter. Finally, at lines 165-172, discoverUrls returns only the first maxPages URLs from the set, along with the raw robots.txt body for later re-parsing.

How robots.txt Enforcement Works

During the crawl phase, every potential link is validated against the stored robots.txt rules before entering the crawl queue.

Pre-Crawl Validation with shouldQueueCrawlLink

The shouldQueueCrawlLink function in src/server/workflows/siteAuditWorkflowCrawl.ts implements a five-point validation gate:

function shouldQueueCrawlLink(
  link: string,
  origin: string,
  robots: RobotsResult,
  visited: Set<string>,
  queued: Set<string>,
): boolean {
  return (
    isSameOrigin(link, origin) &&
    isCrawlableUrl(link) &&
    robots.isAllowed(link) &&      // robots.txt check
    !visited.has(link) &&
    !queued.has(link)
  );
}

This function checks same-origin policy using isSameOrigin from src/server/lib/audit/url-utils.ts, crawlability via isCrawlableUrl from src/server/lib/audit/url-policy.ts, and robots.txt compliance through robots.isAllowed(link). The isAllowed method respects User-agent: * directives and specific agent rules present in the original robots.txt file.

Deterministic Policy Application

The workflow stores the raw robots.txt text in runAuditPhases and re-parses it using parseRobotsTxt before the crawl begins. This ensures that retries of failed steps use the original policy set, not a freshly fetched version that might have changed. The same validation runs inside selectNextCrawlBatch as a duplicate guard, ensuring blocked URLs never reach the page-analysis stage.

Implementation Examples

Discovering URLs during the audit initialization:

import { discoverUrls } from "@/server/lib/audit/discovery";

async function getSeedUrls(origin: string, maxPages: number) {
  const { urls, robotsText } = await discoverUrls(origin, maxPages);
  console.log("Seed URLs:", urls);
  console.log("Raw robots.txt:", robotsText);
}

This corresponds to the discoverUrls implementation in src/server/lib/audit/discovery.ts.

Enforcing robots.txt when queuing links:

import { parseRobotsTxt } from "@/server/lib/audit/discovery";
import { shouldQueueCrawlLink } from "@/server/workflows/siteAuditWorkflowCrawl";

// Using the raw robots.txt from discovery phase
const robots = parseRobotsTxt(origin, robotsText);

if (shouldQueueCrawlLink(link, origin, robots, visitedSet, queuedSet)) {
  // Safe to crawl
  linkQueue.push({ url: link, depth: nextDepth });
}

This pattern uses shouldQueueCrawlLink from src/server/workflows/siteAuditWorkflowCrawl.ts and parseRobotsTxt from src/server/lib/audit/discovery.ts.

Summary

• Two-phase architecture separates URL discovery from crawling, ensuring deterministic behavior across workflow retries.
• Recursive sitemap parsing in src/server/lib/audit/discovery.ts respects depth limits (3 levels) and document counts (300 max) while fetching with 15-second timeouts.
• Deterministic robots enforcement uses the stored raw robots.txt text from discovery, re-parsed before crawling to ensure consistent policy application.
• Multi-layer validation via shouldQueueCrawlLink checks origin, crawlability, and robots.txt rules before queuing any URL.
• Fault tolerance handles missing robots.txt files by defaulting to allow-all policies, ensuring audits complete even with configuration errors.

Frequently Asked Questions

How does Open-SEO handle missing or unreachable robots.txt files?

When fetchRobotsTxtText encounters a network error or non-OK status at lines 29-38 of src/server/lib/audit/discovery.ts, it returns null. The parseRobotsTxt function then generates a default permissive policy where isAllowed always returns true and the sitemap list is empty. This allows the audit to proceed using only discovered conventional sitemap.xml locations or supplied seed URLs.

What happens if sitemaps reference nested sitemaps beyond the depth limit?

The fetchSitemapDocumentWithRetry function enforces MAX_SITEMAP_DEPTH (3) and MAX_SITEMAP_DOCS (300) as hard limits. When these thresholds are reached, the recursion stops gracefully, and the system processes the URLs already collected. This prevents infinite loops on circular sitemap references while maximizing coverage within safe bounds.

Does the audit system respect User-agent specific rules in robots.txt?

Yes. The robots-parser library used in parseRobotsTxt honors User-agent directives. The isAllowed method checks against the specific user-agent string configured for the audit (or defaults to *), blocking access to paths disallowed for that specific agent while permitting access allowed only to other agents.

How does the system prevent crawling URLs blocked by robots.txt during workflow retries?

The system stores the raw robots.txt body during the initial discovery phase in runAuditPhases. Before each crawl batch, it re-parses this stored text rather than fetching fresh rules. This guarantees that retry attempts use the exact same policy snapshot as the original attempt, preventing state mismatches where a site owner changes robots.txt mid-audit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}