# How Site Audit Handles URL Discovery and Policy Filtering in Open SEO

> Open SEO's site audit discovers and filters URLs using sitemaps, robots.txt, and same-origin policies. Learn how it enforces directives before crawling.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-07-31

---

**Open SEO's site audit discovers URLs by recursively fetching sitemaps declared in robots.txt, then filters every candidate through a strict policy layer that enforces same-origin constraints and robots.txt directives before crawling.**

The site audit feature in the `every-app/open-seo` repository implements a deterministic two-stage architecture that separates broad URL discovery from selective policy enforcement. This approach ensures the crawler identifies every potentially crawlable page while strictly adhering to site owner permissions and preventing accidental cross-origin requests.

## URL Discovery: Parsing robots.txt and Sitemaps

The discovery phase begins in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), where the `discoverUrls` function orchestrates the entire process. It first retrieves the target site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) using `fetchRobotsTxtText`, which applies a **10-second timeout** and identifies itself with the custom **OpenSEO-Audit** user-agent string.

The `parseRobotsTxt` function then constructs a `robots-parser` instance, returning both an `isAllowed` helper function and an array of `sitemapUrls` extracted from the file. This parser handles all robots.txt directives, allowing the audit to respect allow/disallow rules throughout the crawl.

### Recursive Sitemap Processing with Safety Limits

Open SEO recursively fetches sitemap files to build the complete URL set, implementing strict guardrails to prevent runaway processing:

- **MAX_SITEMAP_DEPTH = 3** – Limits recursive sitemap index nesting
- **MAX_SITEMAP_DOCS = 300** – Caps total URLs extracted per audit
- **SITEMAP_CONCURRENCY = 5** – Controls parallel fetch operations
- **SITEMAP_RETRIES = 1** – Provides resilience for transient failures

The system uses **fast-xml-parser** configured to treat `<sitemap>` and `<url>` elements as arrays, ensuring consistent parsing regardless of document structure. The helper function `getSitemapLocations` extracts the `<loc>` strings from parsed XML, and every discovered URL passes through `normalizeUrl` (from [`url-utils.ts`](https://github.com/every-app/open-seo/blob/main/url-utils.ts)) to create a canonical representation before storage.

## Policy Filtering: Enforcing Crawl Constraints

Once URLs are discovered, the audit applies policy filtering through `isCrawlableUrl` in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts). This function acts as a gatekeeper, ensuring only appropriate URLs enter the crawl queue.

The policy filter performs three critical checks in sequence:

1. **Same-origin verification** – Uses `isSameOrigin` from [`url-utils.ts`](https://github.com/every-app/open-seo/blob/main/url-utils.ts) to verify the URL shares the identical scheme, host, and port as the audit's start URL, preventing cross-site crawling.
2. **robots.txt compliance** – Queries the parser's `isAllowed` method to confirm the path isn't explicitly disallowed by the site's robots.txt directives.
3. **Content-type validation** – Discards non-HTML resources such as images, CSS files, and binary assets that would waste crawl budget.

All URLs are normalized via `normalizeUrl` before these checks to ensure consistent comparison against policy rules.

## Workflow Integration: Connecting Discovery to Execution

The audit workflow wires these components together across two phases:

- **[`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts)** invokes `discoverUrls` to seed the crawl with the initial URL set derived from sitemaps.
- **[`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts)** iterates over discovered URLs, calling `isCrawlableUrl` for each candidate before invoking page reporters and Lighthouse analysis.

This separation ensures the crawler operates on a deterministic, pre-validated URL list rather than discovering pages ad-hoc during the audit execution.

### Practical Implementation Examples

```typescript
// Starting a site audit with automatic URL discovery
import { AuditService } from "@/server/features/audit/services/AuditService";

async function startAudit(startUrl: string, projectId: string) {
  const audit = await AuditService.create({
    projectId,
    startUrl,
  });

  // Internally runs discoverUrls() and filters via isCrawlableUrl()
  await AuditService.run(audit.id);
}

```

```typescript
// Discovery implementation (simplified from discovery.ts)
export async function discoverUrls(origin: string) {
  const robotsTxt = await fetchRobotsTxtText(origin);
  const robots = parseRobotsTxt(origin, robotsTxt);
  const sitemapUrls = robots.sitemapUrls;

  const crawledUrls = new Set<string>();
  for (const sitemap of sitemapUrls) {
    const pageUrls = await fetchAndParseSitemap(sitemap, 0);
    pageUrls.forEach((u) => crawledUrls.add(normalizeUrl(u)));
  }
  return Array.from(crawledUrls);
}

```

```typescript
// Policy enforcement (simplified from url-policy.ts)
export function isCrawlableUrl(
  url: string, 
  origin: string, 
  robots: RobotsResult
) {
  if (!isSameOrigin(url, origin)) return false;    // Stay on-site
  if (!robots.isAllowed(url)) return false;        // Obey robots.txt
  if (!isHtmlPage(url)) return false;              // Skip non-HTML
  return true;
}

```

## Summary

- **Discovery starts at robots.txt**: The `discoverUrls` function in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) fetches and parses robots.txt to find sitemap declarations.
- **Recursive sitemap fetching respects limits**: The crawler processes sitemaps up to depth 3 with a maximum of 300 documents and 5 concurrent requests.
- **Policy filtering is multi-layered**: `isCrawlableUrl` enforces same-origin rules, robots.txt directives, and HTML-only constraints.
- **Normalization ensures consistency**: All URLs pass through `normalizeUrl` before policy checks to prevent duplicate crawling of equivalent paths.
- **Workflow separation improves reliability**: Discovery runs once during phase initialization, while filtering applies continuously during the crawl execution.

## Frequently Asked Questions

### How does Open SEO find all pages during a site audit?

The audit discovers pages by first fetching the site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file, parsing it to extract sitemap URLs, then recursively fetching those sitemaps up to a depth of three levels. It extracts every `<loc>` entry from the XML, normalizes the URLs, and stores them for policy filtering. This sitemap-driven approach ensures comprehensive coverage of pages the site owner wants indexed.

### What limits prevent sitemap parsing from running indefinitely?

The discovery engine implements four hard limits: a maximum recursion depth of 3 for sitemap indexes, a cap of 300 total documents, concurrency limited to 5 parallel fetches, and a single retry for failed requests. These constants prevent resource exhaustion when processing large or deeply nested sitemap structures.

### How does the audit respect robots.txt directives?

During discovery, `parseRobotsTxt` creates a parser instance that exposes an `isAllowed` function. During the crawl phase, `isCrawlableUrl` invokes this function for every candidate URL, checking against the specific user-agent rules defined in the site's robots.txt. If the parser returns false, the URL is excluded from the crawl queue.

### Why does the policy filter enforce same-origin checks?

The `isSameOrigin` validation ensures the crawler remains within the scheme-host-port boundary of the starting URL, preventing accidental cross-site crawling that could violate security policies or robots.txt intended for different domains. This check runs before robots.txt validation, providing a fast-fail mechanism for URLs outside the audit scope.