# How OpenSEO Defines and Applies URL Policy in Site Audits

> Learn how OpenSEO defines and applies its URL policy in site audits. Discover how it prevents SSRF attacks and blocks private network access for enhanced security.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-09-01

---

**OpenSEO enforces a strict URL policy through centralized validation in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) to prevent SSRF attacks and block access to private networks during site audits.**

The `every-app/open-seo` repository implements a comprehensive OpenSEO URL policy that protects automated site audits from server-side request forgery (SSRF) vulnerabilities and internal network exposure. This policy is defined in a dedicated utility module and applied consistently across the audit lifecycle, from initial URL validation through every link discovered during crawling.

## Core URL Policy Implementation

The OpenSEO URL policy lives in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts), which exports reusable validation functions and security constants. This centralization ensures that both start URLs and discovered links undergo identical security checks.

### Blocked Hosts and Suffixes

The policy maintains explicit blocklists at the top of the file to filter dangerous hostnames before any network request occurs. The `BLOCKED_HOSTS` set includes entries like `localhost`, `metadata.google.internal`, `169.254.169.254`, and `metadata`, while `BLOCKED_HOST_SUFFIXES` covers patterns like `.localhost`, `.local`, and `.internal`【/src/server/lib/audit/url-policy.ts#L3-L15】.

### Private IP Resolution Checks

To catch hostnames that resolve to internal addresses despite appearing safe, the system performs DNS-over-HTTPS (DoH) queries through the `hostnameResolvesToBlockedAddress` function. This validation detects IPv4, IPv6, and IPv4-mapped IPv6 addresses in private ranges such as `10.0.0.0/8` and `fd00::/8`. If resolution reveals a blocked address, the function throws an `AppError` with code `CRAWL_TARGET_BLOCKED`【/src/server/lib/audit/url-policy.ts#L30-L38】.

## URL Policy Enforcement Points

OpenSEO applies its URL policy at two critical stages of the audit workflow, ensuring defense in depth against malicious or misconfigured targets.

### Start URL Validation

When initiating an audit, the user-supplied URL passes through `normalizeAndValidateStartUrl`. This function normalizes the scheme to `http` or `https`, strips URL fragments, validates against blocked host lists, and executes the DoH-based private IP check. Any violation immediately halts the audit with the `CRAWL_TARGET_BLOCKED` error.

### Per-Link Crawl Filtering

During the crawl phase in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), the `shouldQueueCrawlLink` helper filters every discovered link. It invokes `isCrawlableUrl`, which performs synchronous SSRF checks by parsing the URL, enforcing `http:` or `https:` schemes, and reusing the `isBlockedHost` logic from the main policy module【/src/server/lib/audit/url-policy.ts#L96-L107】. Only same-origin URLs that pass these checks and comply with robots.txt are enqueued【/src/server/workflows/siteAuditWorkflowCrawl.ts#L70-L77】.

## Security Protections Provided

The OpenSEO URL policy delivers comprehensive protection through multiple validation layers:

- **Scheme validation** restricts protocols to `http:` and `https:`, preventing `file:`, `data:`, or other dangerous schemes from bypassing network controls.
- **Blocked host detection** stops connections to obvious internal endpoints like `localhost` and cloud metadata services.
- **Suffix-based filtering** catches variations like `api.localhost` or `service.internal` using pattern matching.
- **DoH resolution** identifies hostnames that resolve to private IPs even when the hostname itself appears legitimate.
- **Same-origin enforcement** via `isSameOrigin` prevents the crawler from following redirects to external domains after initial validation.

## Implementation Examples

The following patterns demonstrate how to integrate the OpenSEO URL policy into audit workflows.

Start URL validation:

```typescript
import { normalizeAndValidateStartUrl } from "@/server/lib/audit/url-policy";

async function prepareAudit(rawUrl: string) {
  // Throws AppError("CRAWL_TARGET_BLOCKED") if the URL violates the policy
  const safeUrl = await normalizeAndValidateStartUrl(rawUrl);
  // safeUrl is now a fully-validated, scheme-normalised URL
  return safeUrl;
}

```

Per-link filtering during crawling:

```typescript
import { isCrawlableUrl } from "@/server/lib/audit/url-policy";
import { isSameOrigin } from "@/server/lib/audit/url-utils";

function shouldQueueCrawlLink(link: string, origin: string, robots: any) {
  // Only enqueue if:
  //  – it belongs to the same origin,
  //  – it passes the URL policy check,
  //  – it is allowed by robots.txt
  return (
    isSameOrigin(link, origin) &&
    isCrawlableUrl(link) &&
    robots.isAllowed(link)
  );
}

```

Extending blocked hosts:

```typescript
// In src/server/lib/audit/url-policy.ts, extend BLOCKED_HOSTS:
const BLOCKED_HOSTS = new Set([
  "localhost",
  "metadata.google.internal",
  "metadata",
  "169.254.169.254",
  "100.100.100.200",
  // New entry:
  "internal.example.com",
]);

```

## Summary

- The OpenSEO URL policy is centralized in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) to ensure consistent security validation.
- **SSRF protection** combines static blocklists, suffix matching, and DNS-over-HTTPS resolution to prevent access to private networks.
- Validation occurs at two stages: initial start URL checks via `normalizeAndValidateStartUrl` and per-link filtering via `isCrawlableUrl`.
- The system enforces strict scheme validation (`http`/`https` only) and same-origin policies to maintain crawl boundaries.
- All validation failures throw `AppError` with code `CRAWL_TARGET_BLOCKED` for standardized error handling.

## Frequently Asked Questions

### How does OpenSEO prevent SSRF attacks during site audits?

OpenSEO prevents SSRF through layered validation in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts). The system checks URLs against a blocklist containing `localhost`, cloud metadata endpoints like `169.254.169.254`, and internal host suffixes. Additionally, it performs DNS-over-HTTPS queries to detect hostnames resolving to private IPv4 or IPv6 addresses, throwing `CRAWL_TARGET_BLOCKED` if any check fails.

### Can the URL policy blocklist be customized?

Yes. Developers can modify the `BLOCKED_HOSTS` set and `BLOCKED_HOST_SUFFIXES` array directly in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts). The code uses a simple Set-based structure, allowing straightforward addition of custom internal domains or specific IP addresses that should be excluded from audits.

### What error code is returned when a URL fails policy validation?

When any URL fails the OpenSEO security checks, the system throws an `AppError` with the specific error code `CRAWL_TARGET_BLOCKED`. This standardized code allows calling functions to handle policy violations consistently, whether they occur during initial URL validation or while filtering discovered links during the crawl phase.

### Why does OpenSEO use DNS-over-HTTPS for URL validation?

OpenSEO uses DNS-over-HTTPS (DoH) in the `hostnameResolvesToBlockedAddress` function to prevent DNS rebinding attacks and detect hostnames that resolve to private IP addresses. This approach ensures that even if a hostname appears legitimate (e.g., `safe-looking-domain.com`), the system verifies it does not resolve to internal addresses like `10.0.0.0/8` or `fd00::/8` before allowing the connection.