OpenSEO Sitemap Fetch Limits: Timeout, Size, and Concurrency Boundaries

TLDR: OpenSEO enforces six hard boundaries during sitemap discovery—15‑second fetch timeouts, 10 MiB maximum payload size, 300 URLs per batch, 3‑level crawl depth, 5 concurrent requests, and 1 automatic retry—to protect audit workflows from resource exhaustion.

The every-app/open-seo codebase governs limits for fetching sitemaps through constants defined in src/server/lib/audit/discovery.ts. These constraints prevent oversized XML files, slow network hops, or unbounded URL lists from destabilizing the SEO audit engine.

Hard Limits Defined in discovery.ts

The discovery module declares the following immutable thresholds at the top of the file:

• SITEMAP_FETCH_TIMEOUT_MS = 15_000 – Terminates any HTTP request that stalls beyond 15 seconds ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L8)).
• MAX_SITEMAP_DEPTH = 3 – Restricts how many link levels the crawler follows when seeding from a sitemap entry ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L15)).
• MAX_SITEMAP_DOCS = 300 – Caps the number of URLs extracted from a single sitemap fetch to keep step‑state memory below 1 MiB ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L16)).
• SITEMAP_CONCURRENCY = 5 – Throttles parallel fetches so only five sitemap requests run simultaneously ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L17)).
• SITEMAP_RETRIES = 1 – Grants exactly one retry attempt for transient network failures ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L18)).
• MAX_SITEMAP_BYTES = 10 * 1024 * 1024 – Rejects any response body larger than 10 MiB before parsing ([discovery.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L23)).

How Limits Are Enforced

Fetch Timeout and Retries

When fetchSitemapDocumentWithRetry is invoked, it wraps the HTTP call in an AbortController that triggers after SITEMAP_FETCH_TIMEOUT_MS. If the socket hangs or DNS resolution lags beyond 15 seconds, the promise rejects and the retry logic consumes the single SITEMAP_RETRIES allowance before marking the URL as failed.

Payload Size and Document Count

Incoming streams are piped through a size guard that compares the Content-Length header (or accumulated chunk length) against MAX_SITEMAP_BYTES. Exceeding 10 MiB aborts the download. Once the XML is safely in memory, the parser extracts URLs until it reaches MAX_SITEMAP_DOCS; additional entries are truncated and logged as blocked in the audit output.

Concurrency and Depth

The discovery orchestrator maintains a semaphore limited to SITEMAP_CONCURRENCY. When processing sitemap index files, each nested sitemap inherits the same limits, but the MAX_SITEMAP_DEPTH counter decrements per recursion level, preventing infinite descent into nested indices.

Code Examples

The following snippet demonstrates invoking the discovery utility that internally respects all six limits:

import { discoverSitemapUrls } from "@/server/lib/audit/discovery";

const origin = "https://example.com";
const limit = 100; // Will be clipped to MAX_SITEMAP_DOCS (300) if larger

const { urls, blocked } = await discoverSitemapUrls(origin, { limit });
console.log(`Fetched ${urls.length} URLs from the sitemap`);

To inspect a single sitemap document with retry logic and automatic timeout handling:

import { fetchSitemapDocumentWithRetry } from "@/server/lib/audit/discovery";

const sitemapUrl = "https://example.com/sitemap.xml";
const result = await fetchSitemapDocumentWithRetry(sitemapUrl);
// Respects SITEMAP_CONCURRENCY, timeout, retries, and size limits internally

Related Files in the Audit Pipeline

• src/server/lib/scrape.ts – Consumes the URL array produced by the discovery limits to seed the full site crawl.
• src/server/lib/audit/crawl-window.ts – Applies complementary workflow‑level memory and time budgets that intersect with sitemap fetch constraints.

Summary

• 15‑second timeout aborts stalled sitemap fetches to prevent workflow hangs.
• Single retry (SITEMAP_RETRIES = 1) offers transient fault tolerance without retry storms.
• 10 MiB size cap (MAX_SITEMAP_BYTES) guards against memory exhaustion from oversized XML.
• 300‑URL ceiling (MAX_SITEMAP_DOCS) keeps batch processing lightweight.
• 5‑request concurrency (SITEMAP_CONCURRENCY) throttles network load.
• 3‑level depth (MAX_SITEMAP_DEPTH) blocks runaway recursion in nested sitemap indexes.

Frequently Asked Questions

What happens if a sitemap exceeds the 10 MiB limit?

According to the every-app/open-seo source code, any response body that crosses MAX_SITEMAP_BYTES triggers an immediate abort, and the sitemap is reported as unreachable in the audit results. The parser never attempts to parse partial payloads.

Can I increase the 300‑URL limit for large websites?

The constant MAX_SITEMAP_DOCS is hard‑coded in src/server/lib/audit/discovery.ts and is not exposed through the public API. To process more than 300 URLs, you must batch requests across multiple sitemap files or fork the repository and rebuild with a higher constant.

Why does OpenSEO retry only once?

The SITEMAP_RETRIES = 1 constant balances reliability against audit duration. A single retry catches transient blips (e.g., 502 gateway timeouts) without extending the discovery phase indefinitely. Persistent failures after the second attempt are treated as hard errors.

How does the depth limit affect sitemap index files?

When the crawler encounters a sitemap index (a sitemap containing <sitemap> entries rather than <url> entries), it decrements the internal depth counter. If recursion reaches MAX_SITEMAP_DEPTH, further nested sitemaps are ignored, preventing exponential URL explosion in deeply nested index structures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}