How the OpenSEO Audit URL Policy Filters and Prioritizes Pages: A Complete Technical Guide

The OpenSEO audit URL policy filters pages through layered SSRF protection—including DNS resolution over HTTPS, host blocking, and same-origin enforcement—then prioritizes crawling via seed-driven discovery order and dynamic concurrency adaptation.

The every-app/open-seo repository implements a production-grade site audit crawler that balances security, coverage, and performance. This article explains how its URL policy decides which pages to crawl and in what order, referencing actual source implementations.

---

Filtering: Security-First URL Validation

The audit URL policy applies three distinct filtering layers to prevent server-side request forgery (SSRF), internal network exposure, and cross-origin crawling.

Start URL Validation and Redirect Handling

Before any crawling begins, normalizeAndValidateStartUrl in src/server/lib/audit/url-policy.ts performs comprehensive validation:

• Scheme enforcement: Requires http:// or https://
• Host blocking: Rejects URLs matching BLOCKED_HOSTS or BLOCKED_HOST_SUFFIXES
• Private IP prevention: Uses DNS-over-HTTPS (DoH) resolution to detect internal addresses before connection
• Redirect sanitization: resolveStartUrlRedirects follows up to five hops, re-validating each with identical security checks

// src/server/lib/audit/url-policy.ts
export async function normalizeAndValidateStartUrl(
  rawUrl: string
): Promise<URL> {
  const url = normalizeUrl(rawUrl);
  
  // Block non-HTTP(S) schemes immediately
  if (url.protocol !== 'http:' && url.protocol !== 'https:') {
    throw new AppError('INVALID_URL_SCHEME', 'URL must use HTTP or HTTPS');
  }
  
  // DNS resolution over HTTPS for SSRF protection
  const resolvedIps = await resolveDnsOverHttps(url.hostname);
  for (const ip of resolvedIps) {
    if (isPrivateIp(ip)) {
      throw new AppError('BLOCKED_HOST', 'URL resolves to private IP address');
    }
  }
  
  // Host blocklist check
  if (isBlockedHost(url.hostname)) {
    throw new AppError('BLOCKED_HOST', 'Host is not allowed for crawling');
  }
  
  return url;
}

Mid-Crawl URL Filtering with isCrawlableUrl

Every discovered URL—whether from page links, redirect targets, or sitemap entries—passes through isCrawlableUrl. This synchronous check blocks:

• Non-HTTP(S) schemes (javascript:, data:, file:, etc.)
• IP literals in private ranges (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8)
• Reserved metadata hostnames (localhost, metadata.google.internal, etc.)
• Explicitly blocked hosts and domain suffixes

// src/server/lib/audit/url-policy.ts
export function isCrawlableUrl(href: string): boolean {
  try {
    const url = new URL(href);
    
    // Protocol check
    if (url.protocol !== 'http:' && url.protocol !== 'https:') {
      return false;
    }
    
    // IP literal validation
    if (isIpAddress(url.hostname) && isPrivateIp(url.hostname)) {
      return false;
    }
    
    // Host blocklist
    if (isBlockedHost(url.hostname)) {
      return false;
    }
    
    return true;
  } catch {
    return false; // Invalid URL syntax
  }
}

Same-Origin Boundary Enforcement

After passing SSRF checks, URLs must satisfy isSameOrigin in src/server/lib/audit/url-utils.ts. This function:

• Matches exact hostnames or allows www. prefix variations
• Permits http:// → https:// upgrades on default ports
• Rejects different TLDs, subdomains (unless www-prefixed), and explicit port mismatches

// src/server/lib/audit/url-utils.ts
export function isSameOrigin(startUrl: URL, candidate: URL): boolean {
  const startHost = normalizeHost(startUrl.hostname);
  const candidateHost = normalizeHost(candidate.hostname);
  
  // Exact match or www-prefixed variant
  if (candidateHost === startHost) return true;
  if (`www.${candidateHost}` === startHost) return true;
  if (`www.${startHost}` === candidateHost) return true;
  
  // Safe protocol upgrade
  if (startUrl.protocol === 'http:' && 
      candidate.protocol === 'https:' &&
      startUrl.port === '80' && 
      candidate.port === '443' &&
      candidateHost === startHost) {
    return true;
  }
  
  return false;
}

---

Prioritization: Discovery Order and Adaptive Concurrency

OpenSEO does not assign explicit numerical priorities to individual pages. Instead, it controls crawl ordering through seed-driven discovery and dynamic crawl window adjustment.

Discovery: Robots.txt and Sitemap Seeding

The discoverUrls function in src/server/lib/audit/discovery.ts populates the initial crawl queue with URLs in order of presumed authority:

1. robots.txt parsing: Extracts sitemap directives and crawl-delay hints
2. Sitemap.xml processing: Follows nested sitemap indexes, adds all <loc> entries
3. Fallback sitemap: Requests /sitemap.xml if robots.txt lacks sitemap declarations

These URLs enter a FIFO queue, ensuring sitemap-listed pages (typically the most important) are crawled before discovered links.

// src/server/lib/audit/discovery.ts
export async function* discoverUrls(
  startUrl: URL,
  robotsTxt: RobotsTxt | null
): AsyncGenerator<DiscoveredUrl> {
  const queue: DiscoveredUrl[] = [];
  
  // Priority 1: Sitemaps from robots.txt
  if (robotsTxt?.sitemaps) {
    for (const sitemapUrl of robotsTxt.sitemaps) {
      const urls = await fetchAndParseSitemap(sitemapUrl);
      queue.push(...urls.map(href => ({ 
        href, 
        source: 'robots-txt-sitemap',
        depth: 0 
      })));
    }
  }
  
  // Priority 2: Default sitemap location
  const defaultSitemap = appendPath(startUrl, '/sitemap.xml');
  try {
    const urls = await fetchAndParseSitemap(defaultSitemap.href);
    queue.push(...urls.map(href => ({ 
      href, 
      source: 'default-sitemap',
      depth: 0 
    })));
  } catch {
    // No default sitemap, continue
  }
  
  // Yield for crawl loop
  while (queue.length > 0) {
    yield queue.shift()!;
  }
}

Adaptive Crawl Window: Performance-Based Prioritization

The actual crawling order is shaped by adjustCrawlWindow in src/server/lib/audit/crawl-window.ts. This function dynamically modifies concurrency based on recent page performance:

• Window expands when pages load quickly with HTTP 2xx responses
• Window contracts when encountering slow responses, errors, or blocks
• Memory-safe bounds: Enforces minimum and maximum window sizes

This creates implicit prioritization: well-performing pages complete faster, allowing more of their discovered links to enter the active crawl pool sooner.

// src/server/lib/audit/crawl-window.ts
export const INITIAL_CRAWL_WINDOW = 5;
export const MIN_CRAWL_WINDOW = 2;
export const MAX_CRAWL_WINDOW = 20;
export const CRAWL_WINDOW_ADJUSTMENT_RATE = 0.5;

export function adjustCrawlWindow(
  currentWindow: number,
  recentResults: CrawledPageResult[]
): number {
  const successCount = recentResults.filter(r => 
    r.statusCode >= 200 && r.statusCode < 300 && r.responseTimeMs < 5000
  ).length;
  
  const failureCount = recentResults.length - successCount;
  const successRate = successCount / recentResults.length;
  
  let adjustment = 0;
  
  if (successRate > 0.8) {
    // Strong performance: increase parallelism
    adjustment = Math.ceil(currentWindow * CRAWL_WINDOW_ADJUSTMENT_RATE);
  } else if (successRate < 0.5) {
    // Poor performance: reduce load
    adjustment = -Math.ceil(currentWindow * CRAWL_WINDOW_ADJUSTMENT_RATE);
  }
  
  return clamp(currentWindow + adjustment, MIN_CRAWL_WINDOW, MAX_CRAWL_WINDOW);
}

Depth-Aware Reporting

While depth does not directly affect queue ordering, crawlDepth values propagate through CrawledPageResult types. Deep pages beyond DEEP_PAGE_DEPTH trigger specific issue reporters in src/server/lib/audit/issues/page-reporters.ts—flagging potential crawl traps or infinitely deep navigation patterns without blocking the audit.

---

Complete Usage Example

// src/server/features/audit/startSiteAudit.ts
import { normalizeAndValidateStartUrl } from '@/server/lib/audit/url-policy';
import { isSameOrigin } from '@/server/lib/audit/url-utils';
import { discoverUrls } from '@/server/lib/audit/discovery';
import { adjustCrawlWindow, INITIAL_CRAWL_WINDOW } from '@/server/lib/audit/crawl-window';

async function runSiteAudit(rawStartUrl: string) {
  // Phase 1: Validate and resolve start URL
  const startUrl = await normalizeAndValidateStartUrl(rawStartUrl);
  const resolvedStart = await resolveStartUrlRedirects(startUrl);
  
  // Phase 2: Discover seed URLs
  const robotsTxt = await fetchRobotsTxt(resolvedStart);
  const discovery = discoverUrls(resolvedStart, robotsTxt);
  
  // Phase 3: Crawl with adaptive window
  let windowSize = INITIAL_CRAWL_WINDOW;
  const results: CrawledPageResult[] = [];
  
  let batch: URL[] = [];
  for await (const discovered of discovery) {
    // Filter every candidate
    if (!isCrawlableUrl(discovered.href)) continue;
    if (!isSameOrigin(resolvedStart, new URL(discovered.href))) continue;
    
    batch.push(new URL(discovered.href));
    
    // Process when window is full or discovery exhausted
    if (batch.length >= windowSize || discovery.next === undefined) {
      const batchResults = await crawlBatch(batch);
      results.push(...batchResults);
      
      // Adjust concurrency for next batch
      windowSize = adjustCrawlWindow(windowSize, batchResults);
      batch = [];
    }
  }
  
  return results;
}

---

Summary

• Filtering occurs in three stages: start URL validation with DNS-over-HTTPS SSRF protection, mid-crawl isCrawlableUrl checks for scheme and host safety, and isSameOrigin enforcement for crawl boundary integrity
• Prioritization relies on discovery order (robots.txt → sitemaps → links) and dynamic crawl window adaptation that implicitly favors well-performing pages
• Key implementation files: url-policy.ts for security filtering, url-utils.ts for origin checks, discovery.ts for seed generation, and crawl-window.ts for performance-driven concurrency

---

Frequently Asked Questions

How does OpenSEO prevent SSRF attacks during audits?

The audit URL policy prevents SSRF through DNS-over-HTTPS resolution in normalizeAndValidateStartUrl, which detects private IP addresses before any TCP connection occurs. Redirect chains are followed with identical validation at each hop, and isCrawlableUrl blocks IP literals and internal hostnames at crawl time.

Can the crawler follow links to subdomains?

No. The isSameOrigin function in url-utils.ts strictly enforces hostname matching, allowing only exact matches or www. prefix variations. Different subdomains or TLDs are rejected to maintain crawl boundaries and prevent scope creep.

What happens when a site has no sitemap?

The discoverUrls function falls back to requesting /sitemap.xml at the root path. If neither robots.txt sitemap directives nor the default sitemap exist, the crawl proceeds with only the start URL, discovering additional pages through link following as the audit progresses.

How does the crawl window affect audit speed?

The adaptive crawl window balances speed against reliability. During fast, successful crawling, adjustCrawlWindow increases concurrency up to 20 parallel requests. When encountering slowdowns or errors, it reduces parallelism to as few as 2 requests, preventing resource exhaustion and respecting struggling servers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}