How OpenSEO Defines and Applies URL Policy in Site Audits

OpenSEO enforces a strict URL policy through centralized validation in src/server/lib/audit/url-policy.ts to prevent SSRF attacks and block access to private networks during site audits.

The every-app/open-seo repository implements a comprehensive OpenSEO URL policy that protects automated site audits from server-side request forgery (SSRF) vulnerabilities and internal network exposure. This policy is defined in a dedicated utility module and applied consistently across the audit lifecycle, from initial URL validation through every link discovered during crawling.

Core URL Policy Implementation

The OpenSEO URL policy lives in src/server/lib/audit/url-policy.ts, which exports reusable validation functions and security constants. This centralization ensures that both start URLs and discovered links undergo identical security checks.

Blocked Hosts and Suffixes

The policy maintains explicit blocklists at the top of the file to filter dangerous hostnames before any network request occurs. The BLOCKED_HOSTS set includes entries like localhost, metadata.google.internal, 169.254.169.254, and metadata, while BLOCKED_HOST_SUFFIXES covers patterns like .localhost, .local, and .internal【/src/server/lib/audit/url-policy.ts#L3-L15】.

Private IP Resolution Checks

To catch hostnames that resolve to internal addresses despite appearing safe, the system performs DNS-over-HTTPS (DoH) queries through the hostnameResolvesToBlockedAddress function. This validation detects IPv4, IPv6, and IPv4-mapped IPv6 addresses in private ranges such as 10.0.0.0/8 and fd00::/8. If resolution reveals a blocked address, the function throws an AppError with code CRAWL_TARGET_BLOCKED【/src/server/lib/audit/url-policy.ts#L30-L38】.

URL Policy Enforcement Points

OpenSEO applies its URL policy at two critical stages of the audit workflow, ensuring defense in depth against malicious or misconfigured targets.

Start URL Validation

When initiating an audit, the user-supplied URL passes through normalizeAndValidateStartUrl. This function normalizes the scheme to http or https, strips URL fragments, validates against blocked host lists, and executes the DoH-based private IP check. Any violation immediately halts the audit with the CRAWL_TARGET_BLOCKED error.

Per-Link Crawl Filtering

During the crawl phase in src/server/workflows/siteAuditWorkflowCrawl.ts, the shouldQueueCrawlLink helper filters every discovered link. It invokes isCrawlableUrl, which performs synchronous SSRF checks by parsing the URL, enforcing http: or https: schemes, and reusing the isBlockedHost logic from the main policy module【/src/server/lib/audit/url-policy.ts#L96-L107】. Only same-origin URLs that pass these checks and comply with robots.txt are enqueued【/src/server/workflows/siteAuditWorkflowCrawl.ts#L70-L77】.

Security Protections Provided

The OpenSEO URL policy delivers comprehensive protection through multiple validation layers:

• Scheme validation restricts protocols to http: and https:, preventing file:, data:, or other dangerous schemes from bypassing network controls.
• Blocked host detection stops connections to obvious internal endpoints like localhost and cloud metadata services.
• Suffix-based filtering catches variations like api.localhost or service.internal using pattern matching.
• DoH resolution identifies hostnames that resolve to private IPs even when the hostname itself appears legitimate.
• Same-origin enforcement via isSameOrigin prevents the crawler from following redirects to external domains after initial validation.

Implementation Examples

The following patterns demonstrate how to integrate the OpenSEO URL policy into audit workflows.

Start URL validation:

import { normalizeAndValidateStartUrl } from "@/server/lib/audit/url-policy";

async function prepareAudit(rawUrl: string) {
  // Throws AppError("CRAWL_TARGET_BLOCKED") if the URL violates the policy
  const safeUrl = await normalizeAndValidateStartUrl(rawUrl);
  // safeUrl is now a fully-validated, scheme-normalised URL
  return safeUrl;
}

Per-link filtering during crawling:

import { isCrawlableUrl } from "@/server/lib/audit/url-policy";
import { isSameOrigin } from "@/server/lib/audit/url-utils";

function shouldQueueCrawlLink(link: string, origin: string, robots: any) {
  // Only enqueue if:
  //  – it belongs to the same origin,
  //  – it passes the URL policy check,
  //  – it is allowed by robots.txt
  return (
    isSameOrigin(link, origin) &&
    isCrawlableUrl(link) &&
    robots.isAllowed(link)
  );
}

Extending blocked hosts:

// In src/server/lib/audit/url-policy.ts, extend BLOCKED_HOSTS:
const BLOCKED_HOSTS = new Set([
  "localhost",
  "metadata.google.internal",
  "metadata",
  "169.254.169.254",
  "100.100.100.200",
  // New entry:
  "internal.example.com",
]);

Summary

• The OpenSEO URL policy is centralized in src/server/lib/audit/url-policy.ts to ensure consistent security validation.
• SSRF protection combines static blocklists, suffix matching, and DNS-over-HTTPS resolution to prevent access to private networks.
• Validation occurs at two stages: initial start URL checks via normalizeAndValidateStartUrl and per-link filtering via isCrawlableUrl.
• The system enforces strict scheme validation (http/https only) and same-origin policies to maintain crawl boundaries.
• All validation failures throw AppError with code CRAWL_TARGET_BLOCKED for standardized error handling.

Frequently Asked Questions

How does OpenSEO prevent SSRF attacks during site audits?

OpenSEO prevents SSRF through layered validation in src/server/lib/audit/url-policy.ts. The system checks URLs against a blocklist containing localhost, cloud metadata endpoints like 169.254.169.254, and internal host suffixes. Additionally, it performs DNS-over-HTTPS queries to detect hostnames resolving to private IPv4 or IPv6 addresses, throwing CRAWL_TARGET_BLOCKED if any check fails.

Can the URL policy blocklist be customized?

Yes. Developers can modify the BLOCKED_HOSTS set and BLOCKED_HOST_SUFFIXES array directly in src/server/lib/audit/url-policy.ts. The code uses a simple Set-based structure, allowing straightforward addition of custom internal domains or specific IP addresses that should be excluded from audits.

What error code is returned when a URL fails policy validation?

When any URL fails the OpenSEO security checks, the system throws an AppError with the specific error code CRAWL_TARGET_BLOCKED. This standardized code allows calling functions to handle policy violations consistently, whether they occur during initial URL validation or while filtering discovered links during the crawl phase.

Why does OpenSEO use DNS-over-HTTPS for URL validation?

OpenSEO uses DNS-over-HTTPS (DoH) in the hostnameResolvesToBlockedAddress function to prevent DNS rebinding attacks and detect hostnames that resolve to private IP addresses. This approach ensures that even if a hostname appears legitimate (e.g., safe-looking-domain.com), the system verifies it does not resolve to internal addresses like 10.0.0.0/8 or fd00::/8 before allowing the connection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}