# How the OpenSEO Site Scraper Works: Architecture and Security Features

> Discover how the OpenSEO site scraper works. Learn about its architecture, URL validation, sitemap discovery, and secure bounded fetching for efficient text extraction.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: architecture
- Published: 2026-08-30

---

**The OpenSEO site scraper is a lightweight, dependency-free module that extracts plain-text content from websites through three stages: URL validation with SSRF protection, sitemap-based discovery, and bounded fetching with HTML-to-text conversion.**

The OpenSEO site scraper serves as the core data collection engine in the every-app/open-seo repository, enabling AI agents to consume human-readable snapshots of any website without invoking a headless browser. Implemented in TypeScript within [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts), this tool prioritizes security and resource efficiency through strict network policies and memory-bounded operations. Its architecture ensures that agents receive clean, relevant text content while the system remains protected against malicious inputs.

## URL Validation and SSRF Protection

Every network request begins in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts), where the `normalizeAndValidateStartUrl` function acts as a security gatekeeper. This implementation prevents Server-Side Request Forgery (SSRF) by enforcing four critical checks before any outbound connection proceeds.

### The normalizeAndValidateStartUrl Function

The validation layer performs the following sequence on all URLs—including start URLs, sitemap entries, and redirect targets:

- **Scheme enforcement**: Automatically prepends `https://` if the URL lacks a protocol.
- **Private network blocking**: Immediately rejects localhost references and internal Google metadata endpoints.
- **DNS resolution verification**: Performs DNS-over-HTTPS lookups to confirm the hostname does not resolve to a private IP address.
- **Fragment stripping**: Removes URL fragments to prevent duplicate content processing.

If any check fails, the function throws an `AppError`, halting the operation before a socket connection opens.

## Site Discovery Process

Once validation passes, the `discoverSiteUrls(domain, limit)` function constructs the homepage URL and attempts to fetch `<origin>/sitemap.xml`. This discovery phase balances comprehensiveness with efficiency by capping the number of pages analyzed.

### Parsing Sitemap Entries

When `parseSitemapUrls(xml, origin)` processes the sitemap, it extracts all `<loc>` entries, resolves them against the origin, and filters for same-origin HTML pages. The function explicitly excludes non-HTML XML files and other resources, returning only the homepage plus the capped list of sitemap URLs for further processing.

## Fetching and Text Extraction

For each validated URL, the `fetchText(url)` function initiates a `fetch` request using a custom **User-Agent** header (`OpenSEO-Onboarding/1.0`) and a strict **10-second timeout**. The module handles redirects manually, re-validating each target URL through `normalizeAndValidateStartUrl` before following the chain.

### Bounded Fetching and Memory Safety

To prevent memory exhaustion attacks, the scraper employs `readBoundedText`, which limits the total response size to **2 MiB**. This boundary ensures that massive files or infinite streams cannot crash the service or degrade AI agent performance.

### HTML-to-Text Conversion

The raw HTML undergoes processing by `htmlToText(html)`, which performs aggressive sanitization:

- Strips `<script>`, `<style>`, and `<noscript>` tags
- Removes HTML comments
- Collapses whitespace sequences
- Decodes HTML entities

Finally, `extractTitle(html)` retrieves the page title, and the system preserves only the first **4,000 characters** of text. Pages yielding no readable content are automatically discarded. The resulting object conforms to `{ url, title, text }`.

## Public API Reference

The scraper exposes two primary functions for AI agent integration, both returning an object shaped as `{ pages, blocked }`, where `blocked` indicates whether the site was unreachable or all pages failed validation.

### readSite(domain, maxPages?)

This function orchestrates the full pipeline: discovery followed by text extraction. The `maxPages` parameter controls the upper limit of pages to analyze.

```typescript
// Example: Get a brief text snapshot of a domain (max 5 pages)
import { readSite } from "@/server/lib/scrape";

async function demo() {
  const result = await readSite("example.com");
  if (result.blocked) {
    console.log("Site could not be read.");
    return;
  }
  for (const page of result.pages) {
    console.log(`--- ${page.title ?? "Untitled"} (${page.url})`);
    console.log(page.text);
    console.log("\n");
  }
}
demo();

```

### readPages(urls, maxPages?)

Use this function to analyze specific URLs directly, bypassing automatic discovery. Each URL undergoes the same validation and extraction workflow.

```typescript
// Example: Read a custom list of URLs (e.g., competitor pages)
import { readPages } from "@/server/lib/scrape";

const urls = [
  "https://competitor.com/about",
  "https://competitor.com/blog/post-1",
];
readPages(urls, 2).then(({ pages, blocked }) => {
  console.log(blocked ? "All URLs were blocked" : "Fetched pages:");
  pages.forEach(p => console.log(`${p.title}: ${p.text.slice(0, 200)}…`));
});

```

## Summary

- **Strict URL validation** via `normalizeAndValidateStartUrl` in [`src/server/lib/audit/url-policy.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/url-policy.ts) prevents SSRF attacks by blocking private networks and validating DNS resolution.
- **Sitemap-aware discovery** in `discoverSiteUrls` efficiently identifies crawlable pages while respecting the configured limit.
- **Resource boundaries** include a 2 MiB fetch limit and 10-second timeout to ensure stable operation.
- **Clean text extraction** removes scripts, styles, and markup, returning structured data as `{ url, title, text }` for AI consumption.

## Frequently Asked Questions

### How does the OpenSEO site scraper prevent SSRF attacks?

The scraper validates every URL through `normalizeAndValidateStartUrl`, which blocks private IP ranges, localhost, and internal endpoints while verifying hostnames via DNS-over-HTTPS. Any validation failure triggers an `AppError` before network connection establishment.

### What is the maximum page size the scraper will download?

The `readBoundedText` function enforces a hard limit of **2 MiB** per page. This prevents memory exhaustion when encountering large binaries or maliciously sized responses.

### Can the scraper process websites without a sitemap.xml?

Yes. The `readSite` function always includes the homepage in its analysis. If [`sitemap.xml`](https://github.com/every-app/open-seo/blob/main/sitemap.xml) is absent or unreachable, the scraper returns only the homepage content, provided it passes validation and contains readable text.

### How does the scraper convert HTML to plain text?

The `htmlToText` function strips `<script>`, `<style>`, `<noscript>`, and comment tags, then collapses whitespace and decodes HTML entities. It preserves the first 4,000 characters of the resulting text for AI agent consumption.