What Is the URL Policy System in OpenSEO's Site Crawling for Audits?

The URL policy system in OpenSEO serves as a security gatekeeper that validates start URLs, enforces crawlability rules against robots.txt and private network blocklists, and normalizes URLs before any HTTP request is dispatched during an audit.

OpenSEO (every-app/open-seo) implements a centralized URL policy layer to ensure safe, compliant, and efficient site crawling. Located in src/server/lib/audit/url-policy.ts, this system intercepts every candidate URL before the crawler executes, preventing malformed inputs and unauthorized access to restricted networks.

Start URL Validation and Normalization

Before an audit begins, the policy layer sanitizes user-provided start URLs through the normalizeAndValidateStartUrl function.

This validation performs three critical checks:

• Protocol enforcement: Rejects any scheme other than http or https
• Host resolution verification: Ensures the hostname resolves to a public IP address, blocking localhost and private ranges
• Canonicalization: Strips URL fragments and trailing slashes to prevent duplicate crawling

// src/server/lib/audit/url-policy.ts
export function normalizeAndValidateStartUrl(raw: string): URL {
  const url = new URL(raw.trim());
  if (!/^(http|https)$/.test(url.protocol)) {
    throw new AuditError('INVALID_PROTOCOL');
  }
  // Block private IP ranges, localhost, etc.
  if (isPrivateHost(url.hostname)) {
    throw new AuditError('CRAWL_TARGET_BLOCKED');
  }
  // Strip fragment and trailing slash for canonical form
  url.hash = '';
  url.pathname = url.pathname.replace(/\/+$/, '');
  return url;
}

Enforcing Crawlability Rules

The isCrawlableUrl function determines whether a validated URL may actually be fetched, implementing a hierarchical policy check:

1. Robots.txt compliance: Respects Disallow directives for the wildcard user-agent
2. Blocklist filtering: Excludes private IP ranges and administratively blacklisted hosts
3. Allowlist overrides: Permits explicit domain exceptions for special audit requirements

// src/server/lib/audit/url-policy.ts
export function isCrawlableUrl(
  url: URL,
  { allowlist = [], blocklist = [] }: CrawlRules = {}
): boolean {
  // 1. Respect robots.txt
  if (robotsDisallows(url)) return false;
  // 2. Apply internal blocklist (private IPs, black‑listed hosts)
  if (blocklist.includes(url.hostname) || isPrivateHost(url.hostname)) return false;
  // 3. Allowlist overrides
  if (allowlist.includes(url.hostname)) return true;
  return true;
}

Integration in the Audit Workflow

The URL policy functions integrate directly into the crawl initialization phase within [src/server/workflows/siteAuditWorkflowCrawl.ts](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and the [AuditService.ts](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) orchestrator.

Every audit follows this sequence:

1. Input sanitization: normalizeAndValidateStartUrl processes the raw start URL
2. Policy verification: isCrawlableUrl checks against configured allowlists and robots.txt
3. Error propagation: Failures throw AuditError with code CRAWL_TARGET_BLOCKED
4. User feedback: The [error-messages.ts](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts) client library maps internal codes to actionable UI messages

import {
  normalizeAndValidateStartUrl,
  isCrawlableUrl,
} from '@/server/lib/audit/url-policy';

// Inside the crawl step of SiteAuditWorkflow
const rawStartUrl = input.startUrl;
let startUrl: URL;

try {
  startUrl = normalizeAndValidateStartUrl(rawStartUrl);
} catch (e) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

if (!isCrawlableUrl(startUrl, { allowlist: cfg.allowlist })) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

await crawler.crawl(startUrl);

Security and Privacy Guarantees

The URL policy system prevents OpenSEO from inadvertently scanning internal networks or restricted endpoints. By blocking private IP ranges (RFC 1918) and requiring public DNS resolution before crawling begins, the system ensures compliance with organizational security policies and legal constraints.

Summary

• The URL policy system in OpenSEO centralizes all crawl authorization logic in src/server/lib/audit/url-policy.ts
• normalizeAndValidateStartUrl enforces protocol restrictions, blocks private networks, and canonicalizes URLs
• isCrawlableUrl implements multi-layered checks including robots.txt compliance, administrative blocklists, and optional allowlist overrides
• Integration occurs at the workflow level in siteAuditWorkflowCrawl.ts and AuditService.ts, ensuring consistent enforcement across all audit types
• Failures surface as CRAWL_TARGET_BLOCKED errors, translated to user-friendly messages through the client-side error mapping layer

Frequently Asked Questions

What happens if a URL fails the policy check?

When a URL fails validation, the system throws an AuditError with the code CRAWL_TARGET_BLOCKED. This error propagates through the AuditService and renders in the UI as "This crawl target is blocked by security policy," allowing users to understand why the audit cannot proceed.

How does OpenSEO handle robots.txt restrictions?

The isCrawlableUrl function queries the target's robots.txt file and respects Disallow directives intended for all user-agents. If a path is disallowed, the function returns false before any HTTP request reaches the server.

Can administrators override the default blocklist?

Yes. The policy system accepts an allowlist array in the CrawlRules options passed to isCrawlableUrl. Domains listed in the allowlist bypass standard blocklist checks, enabling audits of otherwise restricted internal staging environments when explicitly authorized.

Where is the URL policy logic tested?

Comprehensive unit tests verifying protocol validation, private IP detection, and allowlist behavior reside in [src/server/lib/audit/url-policy.test.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts), while integration tests in the workflow files ensure proper error propagation through the complete audit pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}