# How the Audit Discovery Process Finds Crawlable Pages in Open‑SEO

> Discover how Open-SEO's audit process identifies crawlable pages by analyzing robots.txt and sitemaps. Learn to optimize your SEO strategy with efficient crawl discovery.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-18

---

**The audit discovery process in Open‑SEO finds crawlable pages by fetching and parsing [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), extracting URL lists from declared sitemaps, and seeding the crawl frontier with only allowed URLs before any actual page fetching begins.**

Open‑SEO's site audit workflow separates URL discovery from content crawling to ensure efficient, standards‑compliant audits. The discovery phase, implemented in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), runs before any pages are fetched and produces a validated seed list that respects site owners' crawling preferences.

## How Open‑SEO Discovery Builds the Crawl Seed List

The discovery process follows a strict precedence to maximize coverage while honoring [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) directives. This design prevents wasted bandwidth on disallowed URLs and ensures large sites can be audited without exhaustive link‑following.

### Step 1: Parse robots.txt for Rules and Sitemaps

The crawler first requests `<origin>/robots.txt`. The `parseRobotsTxt` function extracts three critical pieces of information:

- **Disallow rules** — URL patterns that must be excluded from the crawl
- **Crawl‑Delay** — Throttling directive respected throughout the audit
- **Sitemap URLs** — References to XML sitemap files that enumerate crawlable pages

Any explicit `Disallow` patterns are immediately flagged and excluded from subsequent seed generation.

### Step 2: Fetch and Parse Sitemap Files

Every sitemap URL discovered in [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) (or the fallback `<origin>/sitemap.xml`) is downloaded and parsed. The XML parser collects every `<loc>` entry into a deduplicated set.

The implementation logs discovery metrics at line 304 of [`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts):

```

Sitemap discovery completed … fetched=${fetchedDocs}, failed=${failedDocs},
timedOut=${timedOutDocs}, discoveredUrls=${allUrls.size}

```

This telemetry helps diagnose sitemap availability issues and validates that the seed list represents the site's intended crawlable surface.

### Step 3: Generate the Seed List for the Crawl Phase

URLs gathered from sitemaps become the **seed list** (`seededCount`). If no sitemap is available, the discovery step falls back to a minimal seed set including:

- The homepage (`/`)
- Common path patterns inferred from typical site structures (e.g., `/blog/`, `/products/`)

These URLs are stored in the audit record's `discovery` field and later consumed by [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts).

## Integration with the Audit Workflow

The discovery phase is orchestrated from [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) and designed for resumability and observability.

### Phase State Management

As implemented in [`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts), every audit record tracks its current phase:

```typescript
// audit.schema.ts — currentPhase defaults to "discovery"
currentPhase: z.enum(["discovery", "crawl", "analysis", "report"]).default("discovery"),

```

This state machine ensures that:

1. Discovery runs exactly once per audit
2. Results are cached and reusable if the crawl phase restarts
3. Progress is observable through the database

### Code Example: Orchestrating Discovery and Crawl

```typescript
// Trigger the discovery step inside the audit workflow
const discovery = await runDiscoveryPhase(step, {
  origin,
  auditId,
});

// Parse robots.txt and extract sitemaps
const robots = parseRobotsTxt(origin, discovery.robotsText);

// Use the discovered URLs as the seed for the crawl phase
await runCrawlPhase(step, {
  seedUrls: discovery.urls,   // URLs collected from sitemaps
  robots,
});

```

## Key Design Benefits

Separating discovery from crawling provides three operational advantages:

- **Robots compliance guaranteed** — No disallowed URL ever enters the fetch queue
- **Efficient large‑site handling** — Sitemap‑based seeding avoids the depth‑first explosion of naive link crawling
- **Audit resumability** — Discovery results persist in the database, allowing crawls to restart without redundant sitemap fetching

## Source Code Reference

| File | Purpose |
|------|---------|
| [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) | Core discovery logic — fetches [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), parses sitemaps, builds seed list |
| [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) | Orchestrates audit phases and invokes discovery |
| [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) | Consumes seed URLs from discovery for actual page fetching |
| [`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts) | Stores audit phase state and discovered URL lists |

## Summary

- **Primary entry point**: `runDiscoveryPhase()` in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) initiates the audit discovery process
- **Robots.txt is mandatory** — parsed first to obtain rules and sitemap locations via `parseRobotsTxt()`
- **Sitemaps drive coverage** — all `<loc>` entries from fetched XML files populate the seed list
- **Fallback seeds exist** — homepage and common patterns substitute when sitemaps are unavailable
- **State is persistent** — discovery results cache to `audit.discovery` enabling workflow resumability

## Frequently Asked Questions

### What happens if a site has no sitemap?

The discovery process falls back to a minimal seed set including the homepage and inferred common paths such as `/blog/` or `/products/`. This ensures basic coverage without exhaustive link discovery.

### How does Open‑SEO handle crawl delays specified in robots.txt?

The `Crawl‑Delay` directive is extracted during [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) parsing and enforced throughout both the discovery and crawl phases to respect server capacity constraints.

### Can the discovery phase be restarted if it fails?

Yes. Because discovery results are persisted to the audit record in [`src/db/audit.schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/audit.schema.ts), a failed or interrupted audit can resume without re‑fetching sitemaps or re‑parsing [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt).

### Where is the discovered URL count logged?

Line 304 of [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) emits a structured log entry showing fetched documents, failures, timeouts, and the total unique URLs discovered.