How the OpenSEO Site Scraper Works: Architecture and Security Features

The OpenSEO site scraper is a lightweight, dependency-free module that extracts plain-text content from websites through three stages: URL validation with SSRF protection, sitemap-based discovery, and bounded fetching with HTML-to-text conversion.

The OpenSEO site scraper serves as the core data collection engine in the every-app/open-seo repository, enabling AI agents to consume human-readable snapshots of any website without invoking a headless browser. Implemented in TypeScript within src/server/lib/scrape.ts, this tool prioritizes security and resource efficiency through strict network policies and memory-bounded operations. Its architecture ensures that agents receive clean, relevant text content while the system remains protected against malicious inputs.

URL Validation and SSRF Protection

Every network request begins in src/server/lib/audit/url-policy.ts, where the normalizeAndValidateStartUrl function acts as a security gatekeeper. This implementation prevents Server-Side Request Forgery (SSRF) by enforcing four critical checks before any outbound connection proceeds.

The normalizeAndValidateStartUrl Function

The validation layer performs the following sequence on all URLs—including start URLs, sitemap entries, and redirect targets:

• Scheme enforcement: Automatically prepends https:// if the URL lacks a protocol.
• Private network blocking: Immediately rejects localhost references and internal Google metadata endpoints.
• DNS resolution verification: Performs DNS-over-HTTPS lookups to confirm the hostname does not resolve to a private IP address.
• Fragment stripping: Removes URL fragments to prevent duplicate content processing.

If any check fails, the function throws an AppError, halting the operation before a socket connection opens.

Site Discovery Process

Once validation passes, the discoverSiteUrls(domain, limit) function constructs the homepage URL and attempts to fetch <origin>/sitemap.xml. This discovery phase balances comprehensiveness with efficiency by capping the number of pages analyzed.

Parsing Sitemap Entries

When parseSitemapUrls(xml, origin) processes the sitemap, it extracts all <loc> entries, resolves them against the origin, and filters for same-origin HTML pages. The function explicitly excludes non-HTML XML files and other resources, returning only the homepage plus the capped list of sitemap URLs for further processing.

Fetching and Text Extraction

For each validated URL, the fetchText(url) function initiates a fetch request using a custom User-Agent header (OpenSEO-Onboarding/1.0) and a strict 10-second timeout. The module handles redirects manually, re-validating each target URL through normalizeAndValidateStartUrl before following the chain.

Bounded Fetching and Memory Safety

To prevent memory exhaustion attacks, the scraper employs readBoundedText, which limits the total response size to 2 MiB. This boundary ensures that massive files or infinite streams cannot crash the service or degrade AI agent performance.

HTML-to-Text Conversion

The raw HTML undergoes processing by htmlToText(html), which performs aggressive sanitization:

• Strips <script>, <style>, and <noscript> tags
• Removes HTML comments
• Collapses whitespace sequences
• Decodes HTML entities

Finally, extractTitle(html) retrieves the page title, and the system preserves only the first 4,000 characters of text. Pages yielding no readable content are automatically discarded. The resulting object conforms to { url, title, text }.

Public API Reference

The scraper exposes two primary functions for AI agent integration, both returning an object shaped as { pages, blocked }, where blocked indicates whether the site was unreachable or all pages failed validation.

readSite(domain, maxPages?)

This function orchestrates the full pipeline: discovery followed by text extraction. The maxPages parameter controls the upper limit of pages to analyze.

// Example: Get a brief text snapshot of a domain (max 5 pages)
import { readSite } from "@/server/lib/scrape";

async function demo() {
  const result = await readSite("example.com");
  if (result.blocked) {
    console.log("Site could not be read.");
    return;
  }
  for (const page of result.pages) {
    console.log(`--- ${page.title ?? "Untitled"} (${page.url})`);
    console.log(page.text);
    console.log("\n");
  }
}
demo();

readPages(urls, maxPages?)

Use this function to analyze specific URLs directly, bypassing automatic discovery. Each URL undergoes the same validation and extraction workflow.

// Example: Read a custom list of URLs (e.g., competitor pages)
import { readPages } from "@/server/lib/scrape";

const urls = [
  "https://competitor.com/about",
  "https://competitor.com/blog/post-1",
];
readPages(urls, 2).then(({ pages, blocked }) => {
  console.log(blocked ? "All URLs were blocked" : "Fetched pages:");
  pages.forEach(p => console.log(`${p.title}: ${p.text.slice(0, 200)}…`));
});

Summary

• Strict URL validation via normalizeAndValidateStartUrl in src/server/lib/audit/url-policy.ts prevents SSRF attacks by blocking private networks and validating DNS resolution.
• Sitemap-aware discovery in discoverSiteUrls efficiently identifies crawlable pages while respecting the configured limit.
• Resource boundaries include a 2 MiB fetch limit and 10-second timeout to ensure stable operation.
• Clean text extraction removes scripts, styles, and markup, returning structured data as { url, title, text } for AI consumption.

Frequently Asked Questions

How does the OpenSEO site scraper prevent SSRF attacks?

The scraper validates every URL through normalizeAndValidateStartUrl, which blocks private IP ranges, localhost, and internal endpoints while verifying hostnames via DNS-over-HTTPS. Any validation failure triggers an AppError before network connection establishment.

What is the maximum page size the scraper will download?

The readBoundedText function enforces a hard limit of 2 MiB per page. This prevents memory exhaustion when encountering large binaries or maliciously sized responses.

Can the scraper process websites without a sitemap.xml?

Yes. The readSite function always includes the homepage in its analysis. If sitemap.xml is absent or unreachable, the scraper returns only the homepage content, provided it passes validation and contains readable text.

How does the scraper convert HTML to plain text?

The htmlToText function strips <script>, <style>, <noscript>, and comment tags, then collapses whitespace and decodes HTML entities. It preserves the first 4,000 characters of the resulting text for AI agent consumption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}