# What Is the URL Policy System in OpenSEO's Site Crawling for Audits?

> Understand OpenSEO's URL policy system. It secures start URLs, enforces crawlability rules, and normalizes URLs for effective site auditing and analysis.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-07-23

---

**The URL policy system in OpenSEO serves as a security gatekeeper that validates start URLs, enforces crawlability rules against robots.txt and private network blocklists, and normalizes URLs before any HTTP request is dispatched during an audit.**

OpenSEO (`every-app/open-seo`) implements a centralized URL policy layer to ensure safe, compliant, and efficient site crawling. Located in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts), this system intercepts every candidate URL before the crawler executes, preventing malformed inputs and unauthorized access to restricted networks.

## Start URL Validation and Normalization

Before an audit begins, the policy layer sanitizes user-provided start URLs through the `normalizeAndValidateStartUrl` function.

This validation performs three critical checks:

- **Protocol enforcement**: Rejects any scheme other than `http` or `https`
- **Host resolution verification**: Ensures the hostname resolves to a public IP address, blocking localhost and private ranges
- **Canonicalization**: Strips URL fragments and trailing slashes to prevent duplicate crawling

```typescript
// src/server/lib/audit/url-policy.ts
export function normalizeAndValidateStartUrl(raw: string): URL {
  const url = new URL(raw.trim());
  if (!/^(http|https)$/.test(url.protocol)) {
    throw new AuditError('INVALID_PROTOCOL');
  }
  // Block private IP ranges, localhost, etc.
  if (isPrivateHost(url.hostname)) {
    throw new AuditError('CRAWL_TARGET_BLOCKED');
  }
  // Strip fragment and trailing slash for canonical form
  url.hash = '';
  url.pathname = url.pathname.replace(/\/+$/, '');
  return url;
}

```

## Enforcing Crawlability Rules

The `isCrawlableUrl` function determines whether a validated URL may actually be fetched, implementing a hierarchical policy check:

1. **Robots.txt compliance**: Respects `Disallow` directives for the wildcard user-agent
2. **Blocklist filtering**: Excludes private IP ranges and administratively blacklisted hosts
3. **Allowlist overrides**: Permits explicit domain exceptions for special audit requirements

```typescript
// src/server/lib/audit/url-policy.ts
export function isCrawlableUrl(
  url: URL,
  { allowlist = [], blocklist = [] }: CrawlRules = {}
): boolean {
  // 1. Respect robots.txt
  if (robotsDisallows(url)) return false;
  // 2. Apply internal blocklist (private IPs, black‑listed hosts)
  if (blocklist.includes(url.hostname) || isPrivateHost(url.hostname)) return false;
  // 3. Allowlist overrides
  if (allowlist.includes(url.hostname)) return true;
  return true;
}

```

## Integration in the Audit Workflow

The URL policy functions integrate directly into the crawl initialization phase within [[`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts)](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and the [[`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts)](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) orchestrator.

Every audit follows this sequence:

1. **Input sanitization**: `normalizeAndValidateStartUrl` processes the raw start URL
2. **Policy verification**: `isCrawlableUrl` checks against configured allowlists and robots.txt
3. **Error propagation**: Failures throw `AuditError` with code `CRAWL_TARGET_BLOCKED`
4. **User feedback**: The [[`error-messages.ts`](https://github.com/every-app/open-seo/blob/main/error-messages.ts)](https://github.com/every-app/open-seo/blob/main/src/client/lib/error-messages.ts) client library maps internal codes to actionable UI messages

```typescript
import {
  normalizeAndValidateStartUrl,
  isCrawlableUrl,
} from '@/server/lib/audit/url-policy';

// Inside the crawl step of SiteAuditWorkflow
const rawStartUrl = input.startUrl;
let startUrl: URL;

try {
  startUrl = normalizeAndValidateStartUrl(rawStartUrl);
} catch (e) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

if (!isCrawlableUrl(startUrl, { allowlist: cfg.allowlist })) {
  throw new AuditError('CRAWL_TARGET_BLOCKED');
}

await crawler.crawl(startUrl);

```

## Security and Privacy Guarantees

The URL policy system prevents OpenSEO from inadvertently scanning internal networks or restricted endpoints. By blocking private IP ranges (RFC 1918) and requiring public DNS resolution before crawling begins, the system ensures compliance with organizational security policies and legal constraints.

## Summary

- The **URL policy system** in OpenSEO centralizes all crawl authorization logic in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts)
- **`normalizeAndValidateStartUrl`** enforces protocol restrictions, blocks private networks, and canonicalizes URLs
- **`isCrawlableUrl`** implements multi-layered checks including robots.txt compliance, administrative blocklists, and optional allowlist overrides
- Integration occurs at the workflow level in [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts) and [`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts), ensuring consistent enforcement across all audit types
- Failures surface as `CRAWL_TARGET_BLOCKED` errors, translated to user-friendly messages through the client-side error mapping layer

## Frequently Asked Questions

### What happens if a URL fails the policy check?

When a URL fails validation, the system throws an `AuditError` with the code `CRAWL_TARGET_BLOCKED`. This error propagates through the `AuditService` and renders in the UI as "This crawl target is blocked by security policy," allowing users to understand why the audit cannot proceed.

### How does OpenSEO handle robots.txt restrictions?

The `isCrawlableUrl` function queries the target's robots.txt file and respects `Disallow` directives intended for all user-agents. If a path is disallowed, the function returns `false` before any HTTP request reaches the server.

### Can administrators override the default blocklist?

Yes. The policy system accepts an `allowlist` array in the `CrawlRules` options passed to `isCrawlableUrl`. Domains listed in the allowlist bypass standard blocklist checks, enabling audits of otherwise restricted internal staging environments when explicitly authorized.

### Where is the URL policy logic tested?

Comprehensive unit tests verifying protocol validation, private IP detection, and allowlist behavior reside in [[`src/server/lib/audit/url-policy.test.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.test.ts), while integration tests in the workflow files ensure proper error propagation through the complete audit pipeline.