Understanding the URL Policy System for Site Audits in OpenSEO: Implementation and Usage

The URL policy system in OpenSEO is a centralized validation layer in src/server/lib/audit/url-policy.ts that uses normalizeAndValidateStartUrl() and isCrawlableUrl() to sanitize start URLs, block private hosts, enforce robots.txt compliance, and prevent unauthorized crawling of internal networks.

The URL policy for site audits is a critical security and reliability component in the OpenSEO repository. Before any crawler fetches a page, the policy layer validates the target URL against configurable allowlists, blocklists, and robots.txt directives. This ensures every audit respects site owner preferences and protects the platform from scanning restricted infrastructure.

Core Responsibilities of the URL Policy System

The policy implementation in [src/server/lib/audit/url-policy.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) handles four primary duties:

• Start URL normalization and validation — parses, sanitizes, and validates the user-supplied seed URL
• Crawlability enforcement — consults robots.txt, internal blocklists, and admin allowlists
• Private network protection — blocks localhost and RFC‑1918 IP ranges
• Consistent error propagation — throws typed AuditError codes for downstream handling

Key Functions in url-policy.ts

normalizeAndValidateStartUrl()

This function prepares the entry point for every audit by enforcing protocol rules, stripping fragments, and rejecting unsafe hosts.

// src/server/lib/audit/url-policy.ts
export function normalizeAndValidateStartUrl(raw: string): URL {
  const url = new URL(raw.trim());
  if (!/^(http|https)$/.test(url.protocol)) {
    throw new AuditError('INVALID_PROTOCOL');
  }
  // Block private IP ranges, localhost, etc.
  if (isPrivateHost(url.hostname)) {
    throw new AuditError('CRAWL_TARGET_BLOCKED');
  }
  // Strip fragment and trailing slash for canonical form
  url.hash = '';
  url.pathname = url.pathname.replace(/\/+$/, '');
  return url;
}

The function throws AuditError('INVALID_PROTOCOL') for non-HTTP(S) schemes and AuditError('CRAWL_TARGET_BLOCKED') when the hostname resolves to a private address.

isCrawlableUrl()

The crawlability check runs after normalization and decides whether the crawler may proceed. It respects three ordered rule sources:

1. Robots.txt directives — parsed via robotsDisallows(url) for the * user-agent
2. Internal blocklist — hardcoded private ranges plus runtime blocklist array
3. Admin allowlist — explicit overrides for trusted domains

// src/server/lib/audit/url-policy.ts
export function isCrawlableUrl(
  url: URL,
  { allowlist = [], blocklist = [] }: CrawlRules = {}
): boolean {
  // 1. Respect robots.txt
  if (robotsDisallows(url)) return false;
  // 2. Apply internal blocklist (private IPs, black-listed hosts)
  if (blocklist.includes(url.hostname) || isPrivateHost(url.hostname)) return false;
  // 3. Allowlist overrides
  if (allowlist.includes(url.hostname)) return true;
  return true;
}

Return values are boolean: true permits crawling, false blocks silently (callers typically convert this to CRAWL_TARGET_BLOCKED).

Integration with the Site Audit Workflow

The policy functions are imported in [src/server/workflows/siteAuditWorkflowCrawl.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and invoked before the crawler initializes:

import {
  normalizeAndValidateStartUrl,
  isCrawlableUrl,
} from '@/server/lib/audit/url-policy';

// Inside the crawl step of SiteAuditWorkflow
const rawStartUrl = input.startUrl;                     // e.g. "https://example.com"
let startUrl: URL;

try {
  startUrl = normalizeAndValidateStartUrl(rawStartUrl);
} catch (e) {
  // Propagate a friendly error to the UI
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

// Verify crawlability before launching the crawler
if (!isCrawlableUrl(startUrl, { allowlist: cfg.allowlist })) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

// Proceed with the actual crawling logic …
await crawler.crawl(startUrl);

This pattern ensures no URL reaches the crawler without passing policy gates.

Error Handling and User Messaging

When validation fails, the workflow propagates AuditError codes to [src/server/features/audit/services/AuditService.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts), which maps them for the client. The UI layer in [src/client/lib/error-messages.ts](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts) renders:

"This crawl target is blocked by security policy."

This explicit messaging replaces silent failures and aids debugging.

Testing and Reliability

The policy logic is exercised in [src/server/lib/audit/url-policy.test.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts) with cases covering:

• Malformed URLs and unsupported protocols
• Private IP ranges (10.x.x.x, 192.168.x.x, 127.0.0.1)
• robots.txt disallow patterns
• Allowlist and blocklist precedence

These tests prevent regressions when the crawl rules evolve.

Configuration Options

The CrawlRules interface passed to isCrawlableUrl() supports runtime customization:

Parameter	Type	Purpose
allowlist	string[]	Hostnames exempt from robots.txt and blocklist checks
blocklist	string[]	Additional hostnames to reject beyond private ranges

Admins set these arrays via environment variables or the audit configuration UI.

Summary

• The URL policy system for site audits lives in src/server/lib/audit/url-policy.ts and exposes normalizeAndValidateStartUrl() and isCrawlableUrl()
• Validation enforces HTTP(S) protocols, strips fragments, and canonicalizes paths
• Security blocks private IPs, localhost, and user-defined blacklisted hosts
• Compliance respects robots.txt disallow directives with optional allowlist overrides
• Integration occurs in siteAuditWorkflowCrawl.ts before any network request
• Errors surface as CRAWL_TARGET_BLOCKED with user-friendly messages via AuditService.ts

Frequently Asked Questions

How does OpenSEO prevent crawling of internal company networks?

The isPrivateHost() helper inside url-policy.ts rejects RFC‑1918 ranges (10/8, 172.16/12, 192.168/16) and localhost addresses. This check runs unconditionally in normalizeAndValidateStartUrl() before any external resolution occurs.

Can I whitelist a subdomain that robots.txt blocks?

Yes. Pass the hostname in the allowlist array to isCrawlableUrl(). The function checks allowlist membership after robots.txt and blocklist evaluation, so explicit permits override implicit denials.

What happens if a user submits a URL with a ftp:// scheme?

normalizeAndValidateStartUrl() tests the protocol against /^(http|https)$/ and throws AuditError('INVALID_PROTOCOL'). The workflow catches this and surfaces a validation error in the audit UI.

Where are URL policy errors translated for end users?

The mapping occurs in src/client/lib/error-messages.ts, which converts internal codes like CRAWL_TARGET_BLOCKED into localized strings. [AuditService.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) serializes these for the API response.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}