What Is the Discovery Phase in OpenSEO's Site Audit? A Technical Breakdown

The discovery phase in OpenSEO's site audit is the initial workflow step that collects a reliable set of seed URLs by fetching and parsing robots.txt, extracting sitemap locations, downloading and parsing sitemaps (up to 3 levels deep), and aggregating URLs for later crawling.

The discovery phase serves as the foundation for every site audit in the open-source OpenSEO project. According to the source code in [src/server/lib/audit/discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), this phase ensures the crawler works from a well-defined, origin-restricted URL set before any actual page crawling or SEO analysis begins. Understanding how the discovery phase works helps developers customize audit behavior, debug URL collection issues, and optimize performance for large sites.

How the Discovery Phase Works

The discovery phase executes four sequential operations to build the seed URL list. Each operation includes specific safeguards to prevent resource exhaustion.

1. Fetching and Parsing robots.txt

The phase begins by retrieving the target origin's robots.txt file. The fetchAndParseRobots function handles this with two responsibilities:

• Access control: Determines which URLs are allowed or disallowed for crawling based on user-agent rules
• Sitemap discovery: Extracts any <sitemap> entries declared in the file

The response body is capped at 500 KiB to prevent memory issues from malformed responses.

2. Adding Default Sitemap Location

If robots.txt does not declare any sitemaps, the discovery phase automatically adds /sitemap.xml as a fallback location. This ensures broad compatibility without requiring manual configuration.

3. Downloading and Parsing Sitemaps

The discoverUrls function orchestrates sitemap processing with strict limits:

Limit	Value	Purpose
Max depth	3 levels	Prevents infinite recursion in sitemap indexes
Max documents	300 total	Avoids memory blow-ups from malicious sitemaps
Download cap	10 MiB per sitemap	Protects against oversized XML files
Origin constraint	Same-origin only	Blocks external URL injection

Nested sitemap indexes are supported and traversed recursively until depth or document limits are reached. Invalid XML documents or cross-origin URLs are skipped silently.

4. Aggregating and Truncating URLs

After collecting URLs from all sitemap sources, the discovery phase:

• Deduplicates entries
• Truncates the list to the audit's page budget (default: 50 pages)
• Returns a structured payload with urls and robotsText

The raw robots.txt content is cached for later replay during the crawl phase, ensuring consistent access rule evaluation.

Discovery Phase Integration in the Audit Workflow

The discovery phase connects to the broader site audit through [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts). The workflow:

1. Sets currentPhase to "discovery" in the audit record (schema defined in [src/db/audit.schema.ts](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts))
2. Invokes discoverUrls with the origin and page budget
3. Persists results via [AuditRepository.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)
4. Advances to crawl phase only after successful completion

Code Examples

Running Discovery Standalone

// Example: running the discovery phase for https://example.com
import { discoverUrls } from "@/server/lib/audit/discovery";

async function run() {
  const { urls, robotsText } = await discoverUrls("https://example.com", 50);
  console.log("Seed URLs:", urls);
  console.log("robots.txt (cached):", robotsText);
}
run();

Integration in Site Audit Workflow

// Inside the site‑audit workflow (simplified)
import { discoverUrls, parseRobotsTxt } from "@/server/lib/audit/discovery";

export async function runDiscoveryPhase(step, origin) {
  const discovery = await discoverUrls(origin, step.maxPages);
  const robots = parseRobotsTxt(origin, discovery.robotsText);
  // Store `discovery.urls` as crawl seeds and persist `robots` for later checks
  return { ...step, seeds: discovery.urls, robots };
}

Key Source Files

• [src/server/lib/audit/discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) — Core implementation with discoverUrls, fetchAndParseRobots, and sitemap handling
• [src/server/workflows/siteAuditWorkflowPhases.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) — Orchestrates phase transitions
• [src/db/audit.schema.ts](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts) — Database schema with currentPhase field
• [src/server/features/audit/repositories/AuditRepository.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts) — Persistence layer for discovery results

Summary

• The discovery phase is OpenSEO's mandatory first step for every site audit, implemented in discovery.ts
• It collects seed URLs by parsing robots.txt, discovering sitemaps, and processing XML sitemap files with depth and size limits
• Safety constraints include 3-level depth limit, 300 document maximum, 10 MiB sitemap cap, and 500 KiB robots.txt cap
• Results are truncated to the page budget (default 50) and stored with cached robots.txt content for downstream phases
• The workflow explicitly tracks currentPhase: "discovery" and only proceeds after successful completion

Frequently Asked Questions

What happens if a site has no robots.txt or sitemap?

OpenSEO gracefully handles missing discovery sources. If robots.txt returns 404, the phase immediately falls back to checking /sitemap.xml. If no sitemap exists, the discovery phase completes with an empty URL list, and the audit transitions to crawl with zero seeds—though most audits will flag this as a configuration issue.

Can the discovery phase be customized for larger sites?

Yes. The discoverUrls function accepts a maxPages parameter (default 50) that controls final URL budget truncation. However, the 300-document sitemap limit and 10 MiB download caps are hardcoded constants in discovery.ts—modifying these requires editing the source and rebuilding.

How does OpenSEO prevent crawling URLs blocked by robots.txt?

The discovery phase parses robots.txt rules and caches the raw content, but it does not filter discovered URLs during this phase. URL-level access control enforcement happens in the subsequent crawl phase, which uses the cached robotsText to validate each request against the same rules.

Why does the discovery phase have separate size limits for robots.txt and sitemaps?

The limits reflect different threat models. robots.txt is fetched first and parsed as text—its 500 KiB cap prevents regex-based parsing attacks. Sitemaps contain structured XML with potential nesting—10 MiB accommodates legitimate large sitemaps while still bounding memory per download.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}