OpenSEO Sitemap Analysis: Discovery, Parsing, and Generation Explained

OpenSEO supports comprehensive sitemap analysis including automated discovery via robots.txt, recursive XML parsing, and build-time generation.

The every-app/open-seo repository implements full-stack sitemap handling that powers both automated SEO audits and custom site reading workflows. Whether you need to ingest a competitor's URL structure or generate a sitemap for your own deployment, the library provides purpose-built utilities for each phase of sitemap processing.

Sitemap Discovery and Parsing

Automated Discovery via robots.txt

OpenSEO initiates sitemap analysis by fetching the target domain's robots.txt file and scanning for Sitemap directives. According to the source code in src/server/lib/audit/discovery.ts, lines 84-94, if no sitemap is declared in the robots file, the system automatically falls back to <origin>/sitemap.xml as a default location.

This discovery phase returns both the raw robots text and any located sitemap URLs, preparing the ground for recursive fetching.

Recursive XML Parsing with discoverUrls

Once sitemap locations are identified, the discoverUrls function (defined in src/server/lib/audit/discovery.ts, lines 60-68) recursively fetches each sitemap—including indexed sitemaps that reference other sitemap files. The implementation uses a fast XML parser to extract URL entries while applying two critical filters:

• Non-HTML entries are discarded (e.g., images or video metadata if not needed for the audit)
• URLs are normalized to ensure they remain within the same origin, preventing external domain leakage (lines 140-162)

This produces a clean array of crawlable URLs ready for downstream processing.

Crawl Seed Management and URL Limits

To maintain compatibility with Cloudflare Workers' 1 MiB payload limit, OpenSEO caps the discovered URL list before passing it to the crawl phase. The workflow configuration in src/server/workflows/siteAuditWorkflowPhases.ts (lines 23-38) implements this ceiling, ensuring that even sites with massive sitemap indexes stay within serverless execution constraints.

This optimization prevents memory exhaustion while still capturing the most important pages defined in your sitemap structure.

Reading Pages from Sitemaps

The readSite API

For agents and chat tools that need actual page content—not just URLs—OpenSEO exposes the high-level readSite function in src/server/lib/scrape.ts (lines 89-101). This API first calls discoverSiteUrls to retrieve the homepage plus all sitemap-derived URLs, then streams the pages through readPages.

The result is plain-text content extraction suitable for downstream natural language processing, enabling use cases like automated onboarding flows or SAM (Search Asset Management) chat interfaces.

import { readSite } from "@/server/lib/scrape";

async function fetchSamplePages(domain: string) {
  const result = await readSite(domain);   // defaults to 5 pages
  if (result.blocked) {
    console.warn("Site could not be read.");
    return;
  }
  result.pages.forEach(p => {
    console.log(`📄 ${p.title ?? "Untitled"} – ${p.url}`);
    console.log(p.text.slice(0, 200), "…");
  });
}

Generating Sitemaps at Build Time

OpenSEO doesn't just consume sitemaps—it produces them. The build-time script at web/scripts/generate-sitemap.js (lines 131-144) emits a standards-compliant sitemap.xml from your list of published URLs. This file is then served at https://openseo.so/sitemap.xml and referenced in web/public/robots.txt to guide search engine crawlers.

Execute the generator via npm:

{
  "scripts": {
    "build": "vite build && node scripts/generate-sitemap.js",
    "sitemap": "node scripts/generate-sitemap.js"
  }
}

Running npm run sitemap creates dist/sitemap.xml with a URL set optimized for search-engine consumption.

Summary

• Discovery: Automatically checks robots.txt for sitemap directives, falling back to /sitemap.xml when none are found (src/server/lib/audit/discovery.ts).
• Parsing: The discoverUrls function recursively parses sitemap XML, filters non-HTML entries, and normalizes URLs to the target origin.
• Limits: URL lists are capped to stay within Cloudflare Workers' 1 MiB payload constraints (src/server/workflows/siteAuditWorkflowPhases.ts).
• Reading: The readSite API combines sitemap discovery with page fetching to provide plain-text content for agents (src/server/lib/scrape.ts).
• Generation: Build-time scripts can emit sitemap.xml files for the hosted application (web/scripts/generate-sitemap.js).

Frequently Asked Questions

How does OpenSEO find sitemaps if they are not listed in robots.txt?

If no Sitemap directive exists in the domain's robots.txt file, OpenSEO automatically attempts to fetch https://<origin>/sitemap.xml as a fallback. This default path check is implemented in src/server/lib/audit/discovery.ts lines 84-94, ensuring broad compatibility even with sites that lack proper robots.txt declarations.

What happens if a sitemap contains thousands of URLs?

The system applies a hard cap to the discovered URL list to maintain Cloudflare Workers compatibility. As defined in src/server/workflows/siteAuditWorkflowPhases.ts lines 23-38, the crawl seed is limited to prevent exceeding the 1 MiB payload restriction, prioritizing the first N URLs found in the sitemap index.

Can OpenSEO handle sitemap index files that reference other sitemaps?

Yes. The discoverUrls function in src/server/lib/audit/discovery.ts recursively processes sitemap index files (denoted by <sitemapindex> tags), fetching and parsing each referenced sitemap until all URLs are collected. This handles large sites that split URLs across multiple sitemap files.

Is it possible to generate a sitemap only for specific routes?

The build-time generator in web/scripts/generate-sitemap.js operates on the list of published URLs passed to it. While the example implementation generates a complete sitemap, you can modify the URL filtering logic at lines 131-144 to include only routes matching specific patterns (e.g., /blog/* or /docs/*) before the XML is written to disk.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}