# What Is the Discovery Phase in OpenSEO's Site Audit? A Technical Breakdown

> Understand the discovery phase in OpenSEO's site audit. Learn how seed URLs are collected by parsing robots.txt, sitemaps, and aggregating URLs for technical SEO analysis.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**The discovery phase in OpenSEO's site audit is the initial workflow step that collects a reliable set of seed URLs by fetching and parsing [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), extracting sitemap locations, downloading and parsing sitemaps (up to 3 levels deep), and aggregating URLs for later crawling.**

The discovery phase serves as the foundation for every site audit in the [open-source OpenSEO project](https://github.com/every-app/open-seo). According to the source code in [[`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), this phase ensures the crawler works from a well-defined, origin-restricted URL set before any actual page crawling or SEO analysis begins. Understanding how the discovery phase works helps developers customize audit behavior, debug URL collection issues, and optimize performance for large sites.

## How the Discovery Phase Works

The discovery phase executes four sequential operations to build the seed URL list. Each operation includes specific safeguards to prevent resource exhaustion.

### 1. Fetching and Parsing robots.txt

The phase begins by retrieving the target origin's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file. The [`fetchAndParseRobots`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L40-L68) function handles this with two responsibilities:

- **Access control**: Determines which URLs are allowed or disallowed for crawling based on user-agent rules
- **Sitemap discovery**: Extracts any `<sitemap>` entries declared in the file

The response body is capped at **500 KiB** to prevent memory issues from malformed responses.

### 2. Adding Default Sitemap Location

If [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) does not declare any sitemaps, the discovery phase automatically adds [`/sitemap.xml`](https://github.com/every-app/open-seo/blob/main//sitemap.xml) as a fallback location. This ensures broad compatibility without requiring manual configuration.

### 3. Downloading and Parsing Sitemaps

The [`discoverUrls`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L22-L30) function orchestrates sitemap processing with strict limits:

| Limit | Value | Purpose |
|-------|-------|---------|
| Max depth | 3 levels | Prevents infinite recursion in sitemap indexes |
| Max documents | 300 total | Avoids memory blow-ups from malicious sitemaps |
| Download cap | 10 MiB per sitemap | Protects against oversized XML files |
| Origin constraint | Same-origin only | Blocks external URL injection |

Nested sitemap indexes are supported and traversed recursively until depth or document limits are reached. Invalid XML documents or cross-origin URLs are skipped silently.

### 4. Aggregating and Truncating URLs

After collecting URLs from all sitemap sources, the discovery phase:

- Deduplicates entries
- Truncates the list to the audit's **page budget** (default: 50 pages)
- Returns a structured payload with `urls` and `robotsText`

The raw [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) content is cached for later replay during the crawl phase, ensuring consistent access rule evaluation.

## Discovery Phase Integration in the Audit Workflow

The discovery phase connects to the broader site audit through [[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts). The workflow:

1. Sets `currentPhase` to `"discovery"` in the audit record (schema defined in [[`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts)](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts))
2. Invokes `discoverUrls` with the origin and page budget
3. Persists results via [[`AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/AuditRepository.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)
4. Advances to crawl phase only after successful completion

## Code Examples

### Running Discovery Standalone

```typescript
// Example: running the discovery phase for https://example.com
import { discoverUrls } from "@/server/lib/audit/discovery";

async function run() {
  const { urls, robotsText } = await discoverUrls("https://example.com", 50);
  console.log("Seed URLs:", urls);
  console.log("robots.txt (cached):", robotsText);
}
run();

```

### Integration in Site Audit Workflow

```typescript
// Inside the site‑audit workflow (simplified)
import { discoverUrls, parseRobotsTxt } from "@/server/lib/audit/discovery";

export async function runDiscoveryPhase(step, origin) {
  const discovery = await discoverUrls(origin, step.maxPages);
  const robots = parseRobotsTxt(origin, discovery.robotsText);
  // Store `discovery.urls` as crawl seeds and persist `robots` for later checks
  return { ...step, seeds: discovery.urls, robots };
}

```

## Key Source Files

- **[[`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts)** — Core implementation with `discoverUrls`, `fetchAndParseRobots`, and sitemap handling
- **[[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)** — Orchestrates phase transitions
- **[[`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts)](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts)** — Database schema with `currentPhase` field
- **[[`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)** — Persistence layer for discovery results

## Summary

- The **discovery phase** is OpenSEO's mandatory first step for every site audit, implemented in [`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)
- It collects seed URLs by parsing [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), discovering sitemaps, and processing XML sitemap files with depth and size limits
- **Safety constraints** include 3-level depth limit, 300 document maximum, 10 MiB sitemap cap, and 500 KiB [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) cap
- Results are truncated to the page budget (default 50) and stored with cached [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) content for downstream phases
- The workflow explicitly tracks `currentPhase: "discovery"` and only proceeds after successful completion

## Frequently Asked Questions

### What happens if a site has no robots.txt or sitemap?

OpenSEO gracefully handles missing discovery sources. If [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) returns 404, the phase immediately falls back to checking [`/sitemap.xml`](https://github.com/every-app/open-seo/blob/main//sitemap.xml). If no sitemap exists, the discovery phase completes with an empty URL list, and the audit transitions to crawl with zero seeds—though most audits will flag this as a configuration issue.

### Can the discovery phase be customized for larger sites?

Yes. The `discoverUrls` function accepts a `maxPages` parameter (default 50) that controls final URL budget truncation. However, the **300-document sitemap limit** and **10 MiB download caps** are hardcoded constants in [`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)—modifying these requires editing the source and rebuilding.

### How does OpenSEO prevent crawling URLs blocked by robots.txt?

The discovery phase parses [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) rules and caches the raw content, but **it does not filter discovered URLs** during this phase. URL-level access control enforcement happens in the subsequent crawl phase, which uses the cached `robotsText` to validate each request against the same rules.

### Why does the discovery phase have separate size limits for robots.txt and sitemaps?

The limits reflect different threat models. [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) is fetched first and parsed as text—its **500 KiB cap** prevents regex-based parsing attacks. Sitemaps contain structured XML with potential nesting—**10 MiB** accommodates legitimate large sitemaps while still bounding memory per download.