# Understanding the URL Policy System for Site Audits in OpenSEO: Implementation and Usage

> Discover OpenSEO's URL policy system for site audits. Learn how it validates start URLs, blocks private hosts, and enforces robots.txt for secure and compliant crawling.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-08-09

---

**The URL policy system in OpenSEO is a centralized validation layer in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) that uses `normalizeAndValidateStartUrl()` and `isCrawlableUrl()` to sanitize start URLs, block private hosts, enforce robots.txt compliance, and prevent unauthorized crawling of internal networks.**

The **URL policy for site audits** is a critical security and reliability component in the [OpenSEO](https://github.com/every-app/open-seo) repository. Before any crawler fetches a page, the policy layer validates the target URL against configurable allowlists, blocklists, and robots.txt directives. This ensures every audit respects site owner preferences and protects the platform from scanning restricted infrastructure.

## Core Responsibilities of the URL Policy System

The policy implementation in [[`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) handles four primary duties:

- **Start URL normalization and validation** — parses, sanitizes, and validates the user-supplied seed URL
- **Crawlability enforcement** — consults robots.txt, internal blocklists, and admin allowlists
- **Private network protection** — blocks localhost and RFC‑1918 IP ranges
- **Consistent error propagation** — throws typed `AuditError` codes for downstream handling

## Key Functions in [`url-policy.ts`](https://github.com/every-app/open-seo/blob/main/url-policy.ts)

### normalizeAndValidateStartUrl()

This function prepares the entry point for every audit by enforcing protocol rules, stripping fragments, and rejecting unsafe hosts.

```typescript
// src/server/lib/audit/url-policy.ts
export function normalizeAndValidateStartUrl(raw: string): URL {
  const url = new URL(raw.trim());
  if (!/^(http|https)$/.test(url.protocol)) {
    throw new AuditError('INVALID_PROTOCOL');
  }
  // Block private IP ranges, localhost, etc.
  if (isPrivateHost(url.hostname)) {
    throw new AuditError('CRAWL_TARGET_BLOCKED');
  }
  // Strip fragment and trailing slash for canonical form
  url.hash = '';
  url.pathname = url.pathname.replace(/\/+$/, '');
  return url;
}

```

The function throws **`AuditError('INVALID_PROTOCOL')`** for non-HTTP(S) schemes and **`AuditError('CRAWL_TARGET_BLOCKED')`** when the hostname resolves to a private address.

### isCrawlableUrl()

The **crawlability check** runs after normalization and decides whether the crawler may proceed. It respects three ordered rule sources:

1. **Robots.txt directives** — parsed via `robotsDisallows(url)` for the `*` user-agent
2. **Internal blocklist** — hardcoded private ranges plus runtime blocklist array
3. **Admin allowlist** — explicit overrides for trusted domains

```typescript
// src/server/lib/audit/url-policy.ts
export function isCrawlableUrl(
  url: URL,
  { allowlist = [], blocklist = [] }: CrawlRules = {}
): boolean {
  // 1. Respect robots.txt
  if (robotsDisallows(url)) return false;
  // 2. Apply internal blocklist (private IPs, black-listed hosts)
  if (blocklist.includes(url.hostname) || isPrivateHost(url.hostname)) return false;
  // 3. Allowlist overrides
  if (allowlist.includes(url.hostname)) return true;
  return true;
}

```

Return values are boolean: `true` permits crawling, `false` blocks silently (callers typically convert this to `CRAWL_TARGET_BLOCKED`).

## Integration with the Site Audit Workflow

The policy functions are imported in [[`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and invoked before the crawler initializes:

```typescript
import {
  normalizeAndValidateStartUrl,
  isCrawlableUrl,
} from '@/server/lib/audit/url-policy';

// Inside the crawl step of SiteAuditWorkflow
const rawStartUrl = input.startUrl;                     // e.g. "https://example.com"
let startUrl: URL;

try {
  startUrl = normalizeAndValidateStartUrl(rawStartUrl);
} catch (e) {
  // Propagate a friendly error to the UI
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

// Verify crawlability before launching the crawler
if (!isCrawlableUrl(startUrl, { allowlist: cfg.allowlist })) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

// Proceed with the actual crawling logic …
await crawler.crawl(startUrl);

```

This pattern ensures **no URL reaches the crawler without passing policy gates**.

## Error Handling and User Messaging

When validation fails, the workflow propagates `AuditError` codes to [[`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts), which maps them for the client. The UI layer in [[`src/client/lib/error-messages.ts`](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts)](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts) renders:

> "This crawl target is blocked by security policy."

This explicit messaging replaces silent failures and aids debugging.

## Testing and Reliability

The policy logic is exercised in [[`src/server/lib/audit/url-policy.test.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts) with cases covering:

- Malformed URLs and unsupported protocols
- Private IP ranges (10.x.x.x, 192.168.x.x, 127.0.0.1)
- robots.txt disallow patterns
- Allowlist and blocklist precedence

These tests prevent regressions when the crawl rules evolve.

## Configuration Options

The `CrawlRules` interface passed to `isCrawlableUrl()` supports runtime customization:

| Parameter | Type | Purpose |
|-----------|------|---------|
| `allowlist` | `string[]` | Hostnames exempt from robots.txt and blocklist checks |
| `blocklist` | `string[]` | Additional hostnames to reject beyond private ranges |

Admins set these arrays via environment variables or the audit configuration UI.

## Summary

- The **URL policy system for site audits** lives in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) and exposes `normalizeAndValidateStartUrl()` and `isCrawlableUrl()`
- **Validation** enforces HTTP(S) protocols, strips fragments, and canonicalizes paths
- **Security** blocks private IPs, localhost, and user-defined blacklisted hosts
- **Compliance** respects robots.txt disallow directives with optional allowlist overrides
- **Integration** occurs in [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts) before any network request
- **Errors** surface as `CRAWL_TARGET_BLOCKED` with user-friendly messages via [`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts)

## Frequently Asked Questions

### How does OpenSEO prevent crawling of internal company networks?

The `isPrivateHost()` helper inside [`url-policy.ts`](https://github.com/every-app/open-seo/blob/main/url-policy.ts) rejects RFC‑1918 ranges (10/8, 172.16/12, 192.168/16) and localhost addresses. This check runs unconditionally in `normalizeAndValidateStartUrl()` before any external resolution occurs.

### Can I whitelist a subdomain that robots.txt blocks?

Yes. Pass the hostname in the `allowlist` array to `isCrawlableUrl()`. The function checks allowlist membership after robots.txt and blocklist evaluation, so explicit permits override implicit denials.

### What happens if a user submits a URL with a `ftp://` scheme?

`normalizeAndValidateStartUrl()` tests the protocol against `/^(http|https)$/` and throws `AuditError('INVALID_PROTOCOL')`. The workflow catches this and surfaces a validation error in the audit UI.

### Where are URL policy errors translated for end users?

The mapping occurs in [`src/client/lib/error-messages.ts`](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts), which converts internal codes like `CRAWL_TARGET_BLOCKED` into localized strings. [[`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) serializes these for the API response.