How OpenSEO Protects Against SSRF Vulnerabilities During Site Audits

OpenSEO prevents SSRF attacks by validating all URLs through a dedicated URL policy module that enforces scheme restrictions, maintains a blocked host list, and performs DNS-over-HTTPS resolution to detect private IP addresses before any request leaves the crawler.

OpenSEO is an open-source SEO audit platform that executes its site-audit crawler within a Cloudflare Workers environment. Because users supply arbitrary start URLs and the crawler follows links discovered during execution, the system must rigorously defend against Server-Side Request Forgery (SSRF) attacks that could probe internal networks or metadata services. According to the OpenSEO source code, this protection is centralized in src/server/lib/audit/url-policy.ts and applied at three distinct stages: initial URL validation, per-link crawling decisions, and redirect chain re-validation.

Start-URL Validation with normalizeAndValidateStartUrl

Before any crawl begins, OpenSEO normalizes and vets the user-supplied start URL through the asynchronous function normalizeAndValidateStartUrl (lines 9‑38 in src/server/lib/audit/url-policy.ts). This function acts as the first line of defense, ensuring that the audit cannot be seeded with a malicious internal address.

Scheme and Hostname Filtering

The validator first enforces that the URL scheme is strictly http or https, rejecting any alternative protocols that could facilitate code execution or file system access. It then checks the hostname against a blocked host set defined at the top of the module (lines 3‑9), which includes entries like localhost, GCE metadata IP addresses (169.254.169.254), and other common internal endpoints.

DNS Resolution and Private IP Blocking

To prevent DNS rebinding attacks where a hostname initially resolves to a public IP but later switches to a private one, OpenSEO performs a DNS-over-HTTPS (DoH) lookup using resolveAddressRecords (lines 42‑64) against Cloudflare’s resolver. The resolved addresses are passed to hostnameResolvesToBlockedAddress, which utilizes isPrivateIpv4 and isPrivateIpv6 (lines 35‑55 and 66‑84) to detect addresses within restricted ranges—including mapped IPv6 addresses—before the crawler fetches any content.

Per-Link SSRF Checks During Crawling

While the audit progresses, the crawler discovers links, sitemap entries, and redirect targets that require lightweight, synchronous validation to maintain performance.

The isCrawlableUrl Filter

The exported function isCrawlableUrl (lines 96‑107) parses each discovered URL and immediately rejects non-HTTP schemes. It delegates hostname validation to isBlockedHost, which ensures the target does not appear in the forbidden list and is not a literal private IP address. This check executes synchronously to avoid the latency of DNS queries during high-volume crawling.

Integration in the Crawl Loop

Within the crawl workflow defined in src/server/workflows/siteAuditWorkflowCrawl.ts, the function shouldQueueCrawlLink (lines 70‑78) invokes isCrawlableUrl as a prerequisite for queueing any newly discovered link. This ensures that every URL—whether from an anchor tag, redirect, or sitemap—passes SSRF scrutiny before the fetch worker receives it.

Redirect Chain Re-validation

SSRF attackers often attempt to bypass filters by using a public URL that redirects to an internal address. OpenSEO neutralizes this vector through hop-by-hop validation.

resolveStartUrlRedirects and Hop-by-Hop Security

The function resolveStartUrlRedirects (lines 45‑85) follows up to five redirect hops, calling normalizeAndValidateStartUrl on every intermediate URL in the chain. This guarantees that no redirect can smuggle the audit into a blocked address space, even if the initial URL appeared legitimate. The architecture is documented in the design specification at specs/0009-site-audit-crawl-architecture.md (line 93).

Code Examples

The following patterns demonstrate how OpenSEO integrates these protections into its audit workflow:

// Validate the audit start URL when a user creates an audit
import { normalizeAndValidateStartUrl } from "@/server/lib/audit/url-policy";

async function startAudit(inputUrl: string) {
  const safeUrl = await normalizeAndValidateStartUrl(inputUrl);
  // safeUrl is guaranteed to be HTTP(S) and not point to a blocked host
}

// Guard each discovered link before queuing it for crawling
import { isCrawlableUrl } from "@/server/lib/audit/url-policy";

function shouldQueueCrawlLink(
  link: string,
  origin: string,
  robots: RobotsResult,
) {
  // Only same-origin, allowed by robots.txt, and passes SSRF check
  return isSameOrigin(link, origin) && isCrawlableUrl(link) && robots.isAllowed(link);
}

// Follow redirects with per-hop SSRF revalidation
import { resolveStartUrlRedirects } from "@/server/lib/audit/url-policy";

const finalUrl = await resolveStartUrlRedirects(startUrl);
// finalUrl has survived all redirect-hop checks

Summary

• Start-URL validation in normalizeAndValidateStartUrl blocks non-HTTP schemes, forbidden hostnames, and private IP ranges using DoH resolution before crawling begins.
• Per-link filtering via isCrawlableUrl ensures every discovered URL passes synchronous SSRF checks in the crawl loop (siteAuditWorkflowCrawl.ts).
• Redirect re-validation in resolveStartUrlRedirects applies the same strict rules to every hop in a redirect chain (up to five levels deep).
• Blocked host definitions explicitly blacklist localhost, cloud metadata endpoints, and private IPv4/IPv6 ranges to prevent internal network probing.

Frequently Asked Questions

What is SSRF and why does OpenSEO specifically need protection against it?

Server-Side Request Forgery (SSRF) is a vulnerability where an attacker tricks a server into making requests to unintended destinations, often allowing access to internal services or cloud metadata APIs. OpenSEO specifically requires protection because it fetches arbitrary user-supplied URLs during site audits; without safeguards, an attacker could use the crawler to scan internal networks or retrieve sensitive instance metadata from the Cloudflare Workers environment.

How does OpenSEO prevent DNS rebinding attacks during audits?

OpenSEO prevents DNS rebinding by performing a DNS-over-HTTPS lookup via resolveAddressRecords when validating the start URL. This resolves the hostname to concrete IP addresses before any HTTP request is made, allowing isPrivateIpv4 and isPrivateIpv6 to detect if the resolution points to restricted ranges (lines 35‑84 in url-policy.ts). By validating the resolved IP rather than just the hostname string, the system blocks hostnames that might dynamically switch to internal addresses after validation.

Why does OpenSEO use synchronous checks for discovered links but asynchronous DNS for start URLs?

OpenSEO uses asynchronous DNS resolution (DoH) for start URLs because this is a one-time cost per audit that allows deep validation of the initial entry point. For discovered links, it uses the synchronous isCrawlableUrl check to maintain crawling performance, examining only the hostname string against the blocked list and rejecting literal private IP addresses without performing a network lookup. This design balances security with the high throughput required for large-scale site audits.

What happens if a redirect during crawling points to a private IP address?

If a redirect points to a private IP address, the resolveStartUrlRedirects function (lines 45‑85 in url-policy.ts) detects the violation when it calls normalizeAndValidateStartUrl on each redirect hop. The function throws an error and terminates the redirect following process, preventing the crawler from ever reaching the internal address. This per-hop re-validation ensures that redirect chains cannot be used to bypass initial SSRF filters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}