How Site Audit Handles URL Discovery and Policy Filtering in Open SEO

Open SEO's site audit discovers URLs by recursively fetching sitemaps declared in robots.txt, then filters every candidate through a strict policy layer that enforces same-origin constraints and robots.txt directives before crawling.

The site audit feature in the every-app/open-seo repository implements a deterministic two-stage architecture that separates broad URL discovery from selective policy enforcement. This approach ensures the crawler identifies every potentially crawlable page while strictly adhering to site owner permissions and preventing accidental cross-origin requests.

URL Discovery: Parsing robots.txt and Sitemaps

The discovery phase begins in src/server/lib/audit/discovery.ts, where the discoverUrls function orchestrates the entire process. It first retrieves the target site's robots.txt using fetchRobotsTxtText, which applies a 10-second timeout and identifies itself with the custom OpenSEO-Audit user-agent string.

The parseRobotsTxt function then constructs a robots-parser instance, returning both an isAllowed helper function and an array of sitemapUrls extracted from the file. This parser handles all robots.txt directives, allowing the audit to respect allow/disallow rules throughout the crawl.

Recursive Sitemap Processing with Safety Limits

Open SEO recursively fetches sitemap files to build the complete URL set, implementing strict guardrails to prevent runaway processing:

• MAX_SITEMAP_DEPTH = 3 – Limits recursive sitemap index nesting
• MAX_SITEMAP_DOCS = 300 – Caps total URLs extracted per audit
• SITEMAP_CONCURRENCY = 5 – Controls parallel fetch operations
• SITEMAP_RETRIES = 1 – Provides resilience for transient failures

The system uses fast-xml-parser configured to treat <sitemap> and <url> elements as arrays, ensuring consistent parsing regardless of document structure. The helper function getSitemapLocations extracts the <loc> strings from parsed XML, and every discovered URL passes through normalizeUrl (from url-utils.ts) to create a canonical representation before storage.

Policy Filtering: Enforcing Crawl Constraints

Once URLs are discovered, the audit applies policy filtering through isCrawlableUrl in src/server/lib/audit/url-policy.ts. This function acts as a gatekeeper, ensuring only appropriate URLs enter the crawl queue.

The policy filter performs three critical checks in sequence:

1. Same-origin verification – Uses isSameOrigin from url-utils.ts to verify the URL shares the identical scheme, host, and port as the audit's start URL, preventing cross-site crawling.
2. robots.txt compliance – Queries the parser's isAllowed method to confirm the path isn't explicitly disallowed by the site's robots.txt directives.
3. Content-type validation – Discards non-HTML resources such as images, CSS files, and binary assets that would waste crawl budget.

All URLs are normalized via normalizeUrl before these checks to ensure consistent comparison against policy rules.

Workflow Integration: Connecting Discovery to Execution

The audit workflow wires these components together across two phases:

• siteAuditWorkflowPhases.ts invokes discoverUrls to seed the crawl with the initial URL set derived from sitemaps.
• siteAuditWorkflowCrawl.ts iterates over discovered URLs, calling isCrawlableUrl for each candidate before invoking page reporters and Lighthouse analysis.

This separation ensures the crawler operates on a deterministic, pre-validated URL list rather than discovering pages ad-hoc during the audit execution.

Practical Implementation Examples

// Starting a site audit with automatic URL discovery
import { AuditService } from "@/server/features/audit/services/AuditService";

async function startAudit(startUrl: string, projectId: string) {
  const audit = await AuditService.create({
    projectId,
    startUrl,
  });

  // Internally runs discoverUrls() and filters via isCrawlableUrl()
  await AuditService.run(audit.id);
}

// Discovery implementation (simplified from discovery.ts)
export async function discoverUrls(origin: string) {
  const robotsTxt = await fetchRobotsTxtText(origin);
  const robots = parseRobotsTxt(origin, robotsTxt);
  const sitemapUrls = robots.sitemapUrls;

  const crawledUrls = new Set<string>();
  for (const sitemap of sitemapUrls) {
    const pageUrls = await fetchAndParseSitemap(sitemap, 0);
    pageUrls.forEach((u) => crawledUrls.add(normalizeUrl(u)));
  }
  return Array.from(crawledUrls);
}

// Policy enforcement (simplified from url-policy.ts)
export function isCrawlableUrl(
  url: string, 
  origin: string, 
  robots: RobotsResult
) {
  if (!isSameOrigin(url, origin)) return false;    // Stay on-site
  if (!robots.isAllowed(url)) return false;        // Obey robots.txt
  if (!isHtmlPage(url)) return false;              // Skip non-HTML
  return true;
}

Summary

• Discovery starts at robots.txt: The discoverUrls function in src/server/lib/audit/discovery.ts fetches and parses robots.txt to find sitemap declarations.
• Recursive sitemap fetching respects limits: The crawler processes sitemaps up to depth 3 with a maximum of 300 documents and 5 concurrent requests.
• Policy filtering is multi-layered: isCrawlableUrl enforces same-origin rules, robots.txt directives, and HTML-only constraints.
• Normalization ensures consistency: All URLs pass through normalizeUrl before policy checks to prevent duplicate crawling of equivalent paths.
• Workflow separation improves reliability: Discovery runs once during phase initialization, while filtering applies continuously during the crawl execution.

Frequently Asked Questions

How does Open SEO find all pages during a site audit?

The audit discovers pages by first fetching the site's robots.txt file, parsing it to extract sitemap URLs, then recursively fetching those sitemaps up to a depth of three levels. It extracts every <loc> entry from the XML, normalizes the URLs, and stores them for policy filtering. This sitemap-driven approach ensures comprehensive coverage of pages the site owner wants indexed.

What limits prevent sitemap parsing from running indefinitely?

The discovery engine implements four hard limits: a maximum recursion depth of 3 for sitemap indexes, a cap of 300 total documents, concurrency limited to 5 parallel fetches, and a single retry for failed requests. These constants prevent resource exhaustion when processing large or deeply nested sitemap structures.

How does the audit respect robots.txt directives?

During discovery, parseRobotsTxt creates a parser instance that exposes an isAllowed function. During the crawl phase, isCrawlableUrl invokes this function for every candidate URL, checking against the specific user-agent rules defined in the site's robots.txt. If the parser returns false, the URL is excluded from the crawl queue.

Why does the policy filter enforce same-origin checks?

The isSameOrigin validation ensures the crawler remains within the scheme-host-port boundary of the starting URL, preventing accidental cross-site crawling that could violate security policies or robots.txt intended for different domains. This check runs before robots.txt validation, providing a fast-fail mechanism for URLs outside the audit scope.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}