How OpenSEO Handles Redirects During Site Scraping: A Technical Deep Dive

OpenSEO handles redirects by manually intercepting 30x responses, validating each hop against SSRF guards, and following exactly one redirect to prevent security bypasses while maintaining full audit visibility.

The OpenSEO site audit engine must crawl websites that frequently use HTTP redirects for SEO migrations, URL canonicalization, or regional routing. Understanding how OpenSEO handles redirects during site scraping is critical for security auditors and developers extending the platform, as the implementation balances aggressive safety checks against the need to map complete redirect chains.

Manual Redirect Control in the Fetch Layer

OpenSEO disables automatic redirect following at the HTTP client level to maintain programmatic control over every navigation. In src/server/lib/scrape.ts, the fetchText function initiates requests with redirect: "manual" 【scrape.ts L52‑L60】.

This configuration forces the Fetch API to return 30x status codes as standard responses rather than automatically pursuing the Location header. By intercepting redirects manually, OpenSEO can inspect each hop against the same security policies applied to seed URLs.

SSRF-Protected Redirect Validation

When the scraper encounters a redirect response, it extracts the Location header and resolves it against the original request URL to produce an absolute target. This resolved URL immediately passes through normalizeAndValidateStartUrl 【scrape.ts L62‑L68】.

The validator implements comprehensive Server-Side Request Forgery (SSRF) protections:

• Blocks private IP ranges and internal network destinations
• Performs DNS-over-HTTPS resolution to prevent DNS rebinding attacks
• Rejects malformed or non-HTTP schemes

If validation fails, the redirect is discarded entirely without fetching the destination 【scrape.ts L69‑L71】.

Single-Hop Safety Limit

OpenSEO does not recursively follow redirect chains. After validating the first redirect target, the system performs a second fetch on that destination—again using redirect: "manual"—and returns the response body from that final hop as the page content 【scrape.ts L72‑L79】.

This one-hop limit is a deliberate security constraint. It prevents attackers from crafting multi-hop redirect chains that might bypass SSRF checks through DNS time-of-check/time-of-use (TOCTOU) attacks or protocol downgrade sequences.

Recording and Crawling Redirect Topology

The audit workflow preserves redirect metadata for SEO analysis beyond the immediate fetch operation. In src/server/workflows/site-audit-workflow-helpers.ts, each discovered page records a redirectUrl property when a 30x response is observed 【helpers.ts L66‑L77】.

The crawl frontier treats this redirectUrl as a distinct page node, enqueueing it for separate crawling 【helpers.ts L66‑L78】. This architecture enables the audit engine to detect:

• Redirect chains (multiple sequential hops)
• Redirect loops (circular navigation patterns)
• Canonical URL mismatches (where the redirect target differs from the declared canonical)

During the siteAuditWorkflowCrawl step, the system checks page.redirectUrl and explicitly adds it to the URL crawl set, ensuring the workflow fully explores the site's redirect topology 【crawl.ts L335‑L339】.

Implementation Examples

The following patterns demonstrate redirect-safe fetching using OpenSEO's internal libraries:

// Fetch a page following OpenSEO's single-redirect safety protocol
import { readPages } from "./src/server/lib/scrape";

const result = await readPages(["https://example.com"]);
if (!result.blocked) {
  console.log(result.pages[0].title);
}

// Manually validate a redirect URL before fetching
import { normalizeAndValidateStartUrl } from "./src/server/lib/audit/url-policy";

async function safeRedirect(location: string, base: string) {
  const absolute = new URL(location, base).toString();
  return await normalizeAndValidateStartUrl(absolute);
}

Summary

• Manual interception: OpenSEO uses redirect: "manual" in src/server/lib/scrape.ts to prevent automatic redirect following.
• Per-hop validation: Every redirect target passes through normalizeAndValidateStartUrl to enforce SSRF protections.
• Single-hop constraint: Only one redirect is followed per request to prevent chain-based security bypasses.
• Metadata preservation: Redirects are recorded in site-audit-workflow-helpers.ts and enqueued as separate crawl nodes for complete topology mapping.
• Audit integration: The siteAuditWorkflowCrawl step consumes redirectUrl properties to ensure comprehensive site coverage.

Frequently Asked Questions

Does OpenSEO follow JavaScript-based meta-refresh redirects?

No. OpenSEO's redirect handling in src/server/lib/scrape.ts specifically processes HTTP 30x status codes and Location headers. Meta-refresh tags in HTML responses are handled separately by the HTML parsing layer and are subject to different validation rules.

Why does OpenSEO limit redirects to a single hop instead of following chains?

The one-hop limit prevents Server-Side Request Forgery (SSRF) attacks that exploit time-of-check/time-of-use (TOCTOU) vulnerabilities across multiple DNS resolutions. According to the source code in src/server/lib/scrape.ts, following only one validated hop ensures the security check occurs immediately before the final request without intermediate network states.

How does OpenSEO detect redirect loops during a site audit?

The workflow records the redirectUrl for each crawled page in src/server/workflows/site-audit-workflow-helpers.ts and enqueues that URL as a separate node. When the crawl encounters the same URL multiple times with redirectUrl properties pointing to previously visited nodes, the audit engine identifies the circular reference pattern.

Can the redirect validation rules be customized for internal networks?

The normalizeAndValidateStartUrl function in src/server/lib/audit/url-policy.ts enforces strict SSRF guards that block private IP ranges by default. Modifying these validations would require forking the URL policy module, as the current implementation prioritizes preventing DNS rebinding and internal network probing over internal site auditing flexibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}