# OpenSEO Sitemap Analysis: Discovery, Parsing, and Generation Explained

> Discover how OpenSEO tackles sitemap analysis with automated discovery, recursive XML parsing, and build-time generation. Learn more about every-app/open-seo.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-08-06

---

**OpenSEO supports comprehensive sitemap analysis including automated discovery via robots.txt, recursive XML parsing, and build-time generation.**

The `every-app/open-seo` repository implements full-stack sitemap handling that powers both automated SEO audits and custom site reading workflows. Whether you need to ingest a competitor's URL structure or generate a sitemap for your own deployment, the library provides purpose-built utilities for each phase of sitemap processing.

## Sitemap Discovery and Parsing

### Automated Discovery via robots.txt

OpenSEO initiates sitemap analysis by fetching the target domain's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file and scanning for `Sitemap` directives. According to the source code in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), lines 84-94, if no sitemap is declared in the robots file, the system automatically falls back to `<origin>/sitemap.xml` as a default location.

This discovery phase returns both the raw robots text and any located sitemap URLs, preparing the ground for recursive fetching.

### Recursive XML Parsing with discoverUrls

Once sitemap locations are identified, the `discoverUrls` function (defined in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts), lines 60-68) recursively fetches each sitemap—including indexed sitemaps that reference other sitemap files. The implementation uses a fast XML parser to extract URL entries while applying two critical filters:

- **Non-HTML entries are discarded** (e.g., images or video metadata if not needed for the audit)
- **URLs are normalized** to ensure they remain within the same origin, preventing external domain leakage (lines 140-162)

This produces a clean array of crawlable URLs ready for downstream processing.

## Crawl Seed Management and URL Limits

To maintain compatibility with Cloudflare Workers' 1 MiB payload limit, OpenSEO caps the discovered URL list before passing it to the crawl phase. The workflow configuration in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) (lines 23-38) implements this ceiling, ensuring that even sites with massive sitemap indexes stay within serverless execution constraints.

This optimization prevents memory exhaustion while still capturing the most important pages defined in your sitemap structure.

## Reading Pages from Sitemaps

### The readSite API

For agents and chat tools that need actual page content—not just URLs—OpenSEO exposes the high-level `readSite` function in [`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts) (lines 89-101). This API first calls `discoverSiteUrls` to retrieve the homepage plus all sitemap-derived URLs, then streams the pages through `readPages`.

The result is plain-text content extraction suitable for downstream natural language processing, enabling use cases like automated onboarding flows or SAM (Search Asset Management) chat interfaces.

```typescript
import { readSite } from "@/server/lib/scrape";

async function fetchSamplePages(domain: string) {
  const result = await readSite(domain);   // defaults to 5 pages
  if (result.blocked) {
    console.warn("Site could not be read.");
    return;
  }
  result.pages.forEach(p => {
    console.log(`📄 ${p.title ?? "Untitled"} – ${p.url}`);
    console.log(p.text.slice(0, 200), "…");
  });
}

```

## Generating Sitemaps at Build Time

OpenSEO doesn't just consume sitemaps—it produces them. The build-time script at [`web/scripts/generate-sitemap.js`](https://github.com/every-app/open-seo/blob/main/web/scripts/generate-sitemap.js) (lines 131-144) emits a standards-compliant [`sitemap.xml`](https://github.com/every-app/open-seo/blob/main/sitemap.xml) from your list of published URLs. This file is then served at `https://openseo.so/sitemap.xml` and referenced in [`web/public/robots.txt`](https://github.com/every-app/open-seo/blob/main/web/public/robots.txt) to guide search engine crawlers.

Execute the generator via npm:

```json
{
  "scripts": {
    "build": "vite build && node scripts/generate-sitemap.js",
    "sitemap": "node scripts/generate-sitemap.js"
  }
}

```

Running `npm run sitemap` creates [`dist/sitemap.xml`](https://github.com/every-app/open-seo/blob/main/dist/sitemap.xml) with a URL set optimized for search-engine consumption.

## Summary

- **Discovery**: Automatically checks [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) for sitemap directives, falling back to [`/sitemap.xml`](https://github.com/every-app/open-seo/blob/main//sitemap.xml) when none are found ([`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts)).
- **Parsing**: The `discoverUrls` function recursively parses sitemap XML, filters non-HTML entries, and normalizes URLs to the target origin.
- **Limits**: URL lists are capped to stay within Cloudflare Workers' 1 MiB payload constraints ([`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)).
- **Reading**: The `readSite` API combines sitemap discovery with page fetching to provide plain-text content for agents ([`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts)).
- **Generation**: Build-time scripts can emit [`sitemap.xml`](https://github.com/every-app/open-seo/blob/main/sitemap.xml) files for the hosted application ([`web/scripts/generate-sitemap.js`](https://github.com/every-app/open-seo/blob/main/web/scripts/generate-sitemap.js)).

## Frequently Asked Questions

### How does OpenSEO find sitemaps if they are not listed in robots.txt?

If no `Sitemap` directive exists in the domain's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file, OpenSEO automatically attempts to fetch `https://<origin>/sitemap.xml` as a fallback. This default path check is implemented in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) lines 84-94, ensuring broad compatibility even with sites that lack proper robots.txt declarations.

### What happens if a sitemap contains thousands of URLs?

The system applies a hard cap to the discovered URL list to maintain Cloudflare Workers compatibility. As defined in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) lines 23-38, the crawl seed is limited to prevent exceeding the 1 MiB payload restriction, prioritizing the first N URLs found in the sitemap index.

### Can OpenSEO handle sitemap index files that reference other sitemaps?

Yes. The `discoverUrls` function in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) recursively processes sitemap index files (denoted by `<sitemapindex>` tags), fetching and parsing each referenced sitemap until all URLs are collected. This handles large sites that split URLs across multiple sitemap files.

### Is it possible to generate a sitemap only for specific routes?

The build-time generator in [`web/scripts/generate-sitemap.js`](https://github.com/every-app/open-seo/blob/main/web/scripts/generate-sitemap.js) operates on the list of published URLs passed to it. While the example implementation generates a complete sitemap, you can modify the URL filtering logic at lines 131-144 to include only routes matching specific patterns (e.g., `/blog/*` or `/docs/*`) before the XML is written to disk.