# How the Open-SEO Audit System Discovers URLs and Enforces robots.txt Policies

> Learn how the Open-SEO audit system discovers URLs by fetching sitemaps and enforces robots.txt policies by validating links before crawling. Optimize your site's accessibility.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-07-30

---

**The Open-SEO audit system discovers URLs by recursively fetching sitemaps declared in robots.txt and enforces policies by validating every candidate link against parsed robots rules before queuing it for crawling.**

The every-app/open-seo repository implements a deterministic two-phase audit workflow that completely separates URL discovery from the actual crawling process. This architecture ensures that **robots.txt** policies are fetched and parsed before any crawling begins, then applied consistently to every URL candidate during the crawl phase.

## How URL Discovery Works

The discovery phase runs once at the start of each audit via `runDiscoveryPhase`, gathering all candidate URLs before any network requests to actual pages occur.

### Fetching and Parsing robots.txt

The process begins in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) where the `discoverUrls` function orchestrates the initial fetch. At lines 29-38, `fetchRobotsTxtText(origin)` performs a GET request to `<origin>/robots.txt` with a **10-second timeout**. If the request fails or returns a non-OK status, the function returns `null` rather than throwing.

At lines 44-58, `parseRobotsTxt(origin, robotsText)` processes the fetched text using the **robots-parser** library. When the input is `null`, it returns a permissive object where `isAllowed` always returns `true` and the sitemap list is empty. This ensures the audit can continue even when robots.txt is unreachable.

### Recursive Sitemap Processing

The system extracts `<sitemap>` locations from the parsed robots.txt and stores them in `robots.sitemapUrls` (lines 91-94). It also automatically appends the conventional `origin + '/sitemap.xml'` as a fallback location.

The `fetchSitemapDocumentWithRetry` function (lines 61-78) handles the actual fetching with a **15-second timeout**. It validates that responses contain XML, then parses them using **fast-xml-parser** to extract both nested sitemap references and actual page URLs. The recursion respects `MAX_SITEMAP_DEPTH` (set to 3) and `MAX_SITEMAP_DOCS` (set to 300), preventing infinite loops on circular references.

### URL Collection and Limits

All discovered page URLs are deduplicated using a JavaScript `Set` named `allUrls`. The system calculates a `maxDiscoveredUrls` limit between **500 and 50,000** based on the `maxPages` parameter. Finally, at lines 165-172, `discoverUrls` returns only the first `maxPages` URLs from the set, along with the raw [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) body for later re-parsing.

## How robots.txt Enforcement Works

During the crawl phase, every potential link is validated against the stored robots.txt rules before entering the crawl queue.

### Pre-Crawl Validation with shouldQueueCrawlLink

The `shouldQueueCrawlLink` function in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) implements a five-point validation gate:

```ts
function shouldQueueCrawlLink(
  link: string,
  origin: string,
  robots: RobotsResult,
  visited: Set<string>,
  queued: Set<string>,
): boolean {
  return (
    isSameOrigin(link, origin) &&
    isCrawlableUrl(link) &&
    robots.isAllowed(link) &&      // robots.txt check
    !visited.has(link) &&
    !queued.has(link)
  );
}

```

This function checks **same-origin policy** using `isSameOrigin` from [`src/server/lib/audit/url-utils.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-utils.ts), **crawlability** via `isCrawlableUrl` from [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts), and **robots.txt compliance** through `robots.isAllowed(link)`. The `isAllowed` method respects `User-agent: *` directives and specific agent rules present in the original robots.txt file.

### Deterministic Policy Application

The workflow stores the raw [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) text in `runAuditPhases` and re-parses it using `parseRobotsTxt` before the crawl begins. This ensures that retries of failed steps use the **original policy set**, not a freshly fetched version that might have changed. The same validation runs inside `selectNextCrawlBatch` as a duplicate guard, ensuring blocked URLs never reach the page-analysis stage.

## Implementation Examples

**Discovering URLs during the audit initialization:**

```ts
import { discoverUrls } from "@/server/lib/audit/discovery";

async function getSeedUrls(origin: string, maxPages: number) {
  const { urls, robotsText } = await discoverUrls(origin, maxPages);
  console.log("Seed URLs:", urls);
  console.log("Raw robots.txt:", robotsText);
}

```

This corresponds to the `discoverUrls` implementation in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts).

**Enforcing robots.txt when queuing links:**

```ts
import { parseRobotsTxt } from "@/server/lib/audit/discovery";
import { shouldQueueCrawlLink } from "@/server/workflows/siteAuditWorkflowCrawl";

// Using the raw robots.txt from discovery phase
const robots = parseRobotsTxt(origin, robotsText);

if (shouldQueueCrawlLink(link, origin, robots, visitedSet, queuedSet)) {
  // Safe to crawl
  linkQueue.push({ url: link, depth: nextDepth });
}

```

This pattern uses `shouldQueueCrawlLink` from [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and `parseRobotsTxt` from [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts).

## Summary

- **Two-phase architecture** separates URL discovery from crawling, ensuring deterministic behavior across workflow retries.
- **Recursive sitemap parsing** in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) respects depth limits (3 levels) and document counts (300 max) while fetching with 15-second timeouts.
- **Deterministic robots enforcement** uses the stored raw robots.txt text from discovery, re-parsed before crawling to ensure consistent policy application.
- **Multi-layer validation** via `shouldQueueCrawlLink` checks origin, crawlability, and robots.txt rules before queuing any URL.
- **Fault tolerance** handles missing robots.txt files by defaulting to allow-all policies, ensuring audits complete even with configuration errors.

## Frequently Asked Questions

### How does Open-SEO handle missing or unreachable robots.txt files?

When `fetchRobotsTxtText` encounters a network error or non-OK status at lines 29-38 of [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), it returns `null`. The `parseRobotsTxt` function then generates a default permissive policy where `isAllowed` always returns `true` and the sitemap list is empty. This allows the audit to proceed using only discovered conventional sitemap.xml locations or supplied seed URLs.

### What happens if sitemaps reference nested sitemaps beyond the depth limit?

The `fetchSitemapDocumentWithRetry` function enforces `MAX_SITEMAP_DEPTH` (3) and `MAX_SITEMAP_DOCS` (300) as hard limits. When these thresholds are reached, the recursion stops gracefully, and the system processes the URLs already collected. This prevents infinite loops on circular sitemap references while maximizing coverage within safe bounds.

### Does the audit system respect User-agent specific rules in robots.txt?

Yes. The **robots-parser** library used in `parseRobotsTxt` honors `User-agent` directives. The `isAllowed` method checks against the specific user-agent string configured for the audit (or defaults to `*`), blocking access to paths disallowed for that specific agent while permitting access allowed only to other agents.

### How does the system prevent crawling URLs blocked by robots.txt during workflow retries?

The system stores the raw [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) body during the initial discovery phase in `runAuditPhases`. Before each crawl batch, it re-parses this stored text rather than fetching fresh rules. This guarantees that retry attempts use the **exact same policy snapshot** as the original attempt, preventing state mismatches where a site owner changes robots.txt mid-audit.