# How OpenSEO Handles Redirects During Site Scraping: A Technical Deep Dive

> Learn how OpenSEO manages redirects during site scraping. Discover its robust approach to security and auditability for seamless data collection.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-08-30

---

**OpenSEO handles redirects by manually intercepting 30x responses, validating each hop against SSRF guards, and following exactly one redirect to prevent security bypasses while maintaining full audit visibility.**

The OpenSEO site audit engine must crawl websites that frequently use HTTP redirects for SEO migrations, URL canonicalization, or regional routing. Understanding how OpenSEO handles redirects during site scraping is critical for security auditors and developers extending the platform, as the implementation balances aggressive safety checks against the need to map complete redirect chains.

## Manual Redirect Control in the Fetch Layer

OpenSEO disables automatic redirect following at the HTTP client level to maintain programmatic control over every navigation. In [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts), the `fetchText` function initiates requests with `redirect: "manual"` 【scrape.ts L52‑L60】.

This configuration forces the Fetch API to return 30x status codes as standard responses rather than automatically pursuing the `Location` header. By intercepting redirects manually, OpenSEO can inspect each hop against the same security policies applied to seed URLs.

## SSRF-Protected Redirect Validation

When the scraper encounters a redirect response, it extracts the `Location` header and resolves it against the original request URL to produce an absolute target. This resolved URL immediately passes through `normalizeAndValidateStartUrl` 【scrape.ts L62‑L68】.

The validator implements comprehensive **Server-Side Request Forgery (SSRF)** protections:

- Blocks private IP ranges and internal network destinations
- Performs DNS-over-HTTPS resolution to prevent DNS rebinding attacks
- Rejects malformed or non-HTTP schemes

If validation fails, the redirect is discarded entirely without fetching the destination 【scrape.ts L69‑L71】.

## Single-Hop Safety Limit

OpenSEO **does not** recursively follow redirect chains. After validating the first redirect target, the system performs a second fetch on that destination—again using `redirect: "manual"`—and returns the response body from that final hop as the page content 【scrape.ts L72‑L79】.

This **one-hop limit** is a deliberate security constraint. It prevents attackers from crafting multi-hop redirect chains that might bypass SSRF checks through DNS time-of-check/time-of-use (TOCTOU) attacks or protocol downgrade sequences.

## Recording and Crawling Redirect Topology

The audit workflow preserves redirect metadata for SEO analysis beyond the immediate fetch operation. In [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts), each discovered page records a `redirectUrl` property when a 30x response is observed 【helpers.ts L66‑L77】.

The crawl frontier treats this `redirectUrl` as a distinct page node, enqueueing it for separate crawling 【helpers.ts L66‑L78】. This architecture enables the audit engine to detect:

- **Redirect chains** (multiple sequential hops)
- **Redirect loops** (circular navigation patterns)
- **Canonical URL mismatches** (where the redirect target differs from the declared canonical)

During the `siteAuditWorkflowCrawl` step, the system checks `page.redirectUrl` and explicitly adds it to the URL crawl set, ensuring the workflow fully explores the site's redirect topology 【crawl.ts L335‑L339】.

## Implementation Examples

The following patterns demonstrate redirect-safe fetching using OpenSEO's internal libraries:

```typescript
// Fetch a page following OpenSEO's single-redirect safety protocol
import { readPages } from "./src/server/lib/scrape";

const result = await readPages(["https://example.com"]);
if (!result.blocked) {
  console.log(result.pages[0].title);
}

```

```typescript
// Manually validate a redirect URL before fetching
import { normalizeAndValidateStartUrl } from "./src/server/lib/audit/url-policy";

async function safeRedirect(location: string, base: string) {
  const absolute = new URL(location, base).toString();
  return await normalizeAndValidateStartUrl(absolute);
}

```

## Summary

- **Manual interception**: OpenSEO uses `redirect: "manual"` in [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts) to prevent automatic redirect following.
- **Per-hop validation**: Every redirect target passes through `normalizeAndValidateStartUrl` to enforce SSRF protections.
- **Single-hop constraint**: Only one redirect is followed per request to prevent chain-based security bypasses.
- **Metadata preservation**: Redirects are recorded in [`site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/site-audit-workflow-helpers.ts) and enqueued as separate crawl nodes for complete topology mapping.
- **Audit integration**: The `siteAuditWorkflowCrawl` step consumes `redirectUrl` properties to ensure comprehensive site coverage.

## Frequently Asked Questions

### Does OpenSEO follow JavaScript-based meta-refresh redirects?

No. OpenSEO's redirect handling in [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts) specifically processes HTTP 30x status codes and `Location` headers. Meta-refresh tags in HTML responses are handled separately by the HTML parsing layer and are subject to different validation rules.

### Why does OpenSEO limit redirects to a single hop instead of following chains?

The one-hop limit prevents Server-Side Request Forgery (SSRF) attacks that exploit time-of-check/time-of-use (TOCTOU) vulnerabilities across multiple DNS resolutions. According to the source code in [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts), following only one validated hop ensures the security check occurs immediately before the final request without intermediate network states.

### How does OpenSEO detect redirect loops during a site audit?

The workflow records the `redirectUrl` for each crawled page in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) and enqueues that URL as a separate node. When the crawl encounters the same URL multiple times with `redirectUrl` properties pointing to previously visited nodes, the audit engine identifies the circular reference pattern.

### Can the redirect validation rules be customized for internal networks?

The `normalizeAndValidateStartUrl` function in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) enforces strict SSRF guards that block private IP ranges by default. Modifying these validations would require forking the URL policy module, as the current implementation prioritizes preventing DNS rebinding and internal network probing over internal site auditing flexibility.