How the Audit Discovery Process Finds Crawlable Pages in Open‑SEO

The audit discovery process in Open‑SEO finds crawlable pages by fetching and parsing robots.txt, extracting URL lists from declared sitemaps, and seeding the crawl frontier with only allowed URLs before any actual page fetching begins.

Open‑SEO's site audit workflow separates URL discovery from content crawling to ensure efficient, standards‑compliant audits. The discovery phase, implemented in src/server/lib/audit/discovery.ts, runs before any pages are fetched and produces a validated seed list that respects site owners' crawling preferences.

How Open‑SEO Discovery Builds the Crawl Seed List

The discovery process follows a strict precedence to maximize coverage while honoring robots.txt directives. This design prevents wasted bandwidth on disallowed URLs and ensures large sites can be audited without exhaustive link‑following.

Step 1: Parse robots.txt for Rules and Sitemaps

The crawler first requests <origin>/robots.txt. The parseRobotsTxt function extracts three critical pieces of information:

• Disallow rules — URL patterns that must be excluded from the crawl
• Crawl‑Delay — Throttling directive respected throughout the audit
• Sitemap URLs — References to XML sitemap files that enumerate crawlable pages

Any explicit Disallow patterns are immediately flagged and excluded from subsequent seed generation.

Step 2: Fetch and Parse Sitemap Files

Every sitemap URL discovered in robots.txt (or the fallback <origin>/sitemap.xml) is downloaded and parsed. The XML parser collects every <loc> entry into a deduplicated set.

The implementation logs discovery metrics at line 304 of discovery.ts:

Sitemap discovery completed … fetched=${fetchedDocs}, failed=${failedDocs},
timedOut=${timedOutDocs}, discoveredUrls=${allUrls.size}

This telemetry helps diagnose sitemap availability issues and validates that the seed list represents the site's intended crawlable surface.

Step 3: Generate the Seed List for the Crawl Phase

URLs gathered from sitemaps become the seed list (seededCount). If no sitemap is available, the discovery step falls back to a minimal seed set including:

• The homepage (/)
• Common path patterns inferred from typical site structures (e.g., /blog/, /products/)

These URLs are stored in the audit record's discovery field and later consumed by src/server/workflows/siteAuditWorkflowCrawl.ts.

Integration with the Audit Workflow

The discovery phase is orchestrated from src/server/workflows/siteAuditWorkflowPhases.ts and designed for resumability and observability.

Phase State Management

As implemented in src/db/audit.schema.ts, every audit record tracks its current phase:

// audit.schema.ts — currentPhase defaults to "discovery"
currentPhase: z.enum(["discovery", "crawl", "analysis", "report"]).default("discovery"),

This state machine ensures that:

1. Discovery runs exactly once per audit
2. Results are cached and reusable if the crawl phase restarts
3. Progress is observable through the database

Code Example: Orchestrating Discovery and Crawl

// Trigger the discovery step inside the audit workflow
const discovery = await runDiscoveryPhase(step, {
  origin,
  auditId,
});

// Parse robots.txt and extract sitemaps
const robots = parseRobotsTxt(origin, discovery.robotsText);

// Use the discovered URLs as the seed for the crawl phase
await runCrawlPhase(step, {
  seedUrls: discovery.urls,   // URLs collected from sitemaps
  robots,
});

Key Design Benefits

Separating discovery from crawling provides three operational advantages:

• Robots compliance guaranteed — No disallowed URL ever enters the fetch queue
• Efficient large‑site handling — Sitemap‑based seeding avoids the depth‑first explosion of naive link crawling
• Audit resumability — Discovery results persist in the database, allowing crawls to restart without redundant sitemap fetching

Source Code Reference

File	Purpose
src/server/lib/audit/discovery.ts	Core discovery logic — fetches robots.txt, parses sitemaps, builds seed list
src/server/workflows/siteAuditWorkflowPhases.ts	Orchestrates audit phases and invokes discovery
src/server/workflows/siteAuditWorkflowCrawl.ts	Consumes seed URLs from discovery for actual page fetching
src/db/audit.schema.ts	Stores audit phase state and discovered URL lists

Summary

• Primary entry point: runDiscoveryPhase() in src/server/workflows/siteAuditWorkflowPhases.ts initiates the audit discovery process
• Robots.txt is mandatory — parsed first to obtain rules and sitemap locations via parseRobotsTxt()
• Sitemaps drive coverage — all <loc> entries from fetched XML files populate the seed list
• Fallback seeds exist — homepage and common patterns substitute when sitemaps are unavailable
• State is persistent — discovery results cache to audit.discovery enabling workflow resumability

Frequently Asked Questions

What happens if a site has no sitemap?

The discovery process falls back to a minimal seed set including the homepage and inferred common paths such as /blog/ or /products/. This ensures basic coverage without exhaustive link discovery.

How does Open‑SEO handle crawl delays specified in robots.txt?

The Crawl‑Delay directive is extracted during robots.txt parsing and enforced throughout both the discovery and crawl phases to respect server capacity constraints.

Can the discovery phase be restarted if it fails?

Yes. Because discovery results are persisted to the audit record in src/db/audit.schema.ts, a failed or interrupted audit can resume without re‑fetching sitemaps or re‑parsing robots.txt.

Where is the discovered URL count logged?

Line 304 of src/server/lib/audit/discovery.ts emits a structured log entry showing fetched documents, failures, timeouts, and the total unique URLs discovered.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}