# OpenSEO Sitemap Fetch Limits: Timeout, Size, and Concurrency Boundaries

> Discover OpenSEO sitemap fetch limits: 15s timeout, 10MiB size, 300 URLs/batch, 3 levels depth, 5 concurrent requests, and 1 retry. Protect your audits.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**TLDR:** OpenSEO enforces six hard boundaries during sitemap discovery—15‑second fetch timeouts, 10 MiB maximum payload size, 300 URLs per batch, 3‑level crawl depth, 5 concurrent requests, and 1 automatic retry—to protect audit workflows from resource exhaustion.

The `every-app/open-seo` codebase governs **limits for fetching sitemaps** through constants defined in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts). These constraints prevent oversized XML files, slow network hops, or unbounded URL lists from destabilizing the SEO audit engine.

## Hard Limits Defined in discovery.ts

The discovery module declares the following immutable thresholds at the top of the file:

- **`SITEMAP_FETCH_TIMEOUT_MS = 15_000`** – Terminates any HTTP request that stalls beyond 15 seconds ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L8)).
- **`MAX_SITEMAP_DEPTH = 3`** – Restricts how many link levels the crawler follows when seeding from a sitemap entry ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L15)).
- **`MAX_SITEMAP_DOCS = 300`** – Caps the number of URLs extracted from a single sitemap fetch to keep step‑state memory below 1 MiB ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L16)).
- **`SITEMAP_CONCURRENCY = 5`** – Throttles parallel fetches so only five sitemap requests run simultaneously ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L17)).
- **`SITEMAP_RETRIES = 1`** – Grants exactly one retry attempt for transient network failures ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L18)).
- **`MAX_SITEMAP_BYTES = 10 * 1024 * 1024`** – Rejects any response body larger than 10 MiB before parsing ([[`discovery.ts`](https://github.com/every-app/open-seo/blob/main/discovery.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts#L23)).

## How Limits Are Enforced

### Fetch Timeout and Retries

When `fetchSitemapDocumentWithRetry` is invoked, it wraps the HTTP call in an `AbortController` that triggers after `SITEMAP_FETCH_TIMEOUT_MS`. If the socket hangs or DNS resolution lags beyond 15 seconds, the promise rejects and the retry logic consumes the single `SITEMAP_RETRIES` allowance before marking the URL as failed.

### Payload Size and Document Count

Incoming streams are piped through a size guard that compares the `Content-Length` header (or accumulated chunk length) against `MAX_SITEMAP_BYTES`. Exceeding 10 MiB aborts the download. Once the XML is safely in memory, the parser extracts URLs until it reaches `MAX_SITEMAP_DOCS`; additional entries are truncated and logged as `blocked` in the audit output.

### Concurrency and Depth

The discovery orchestrator maintains a semaphore limited to `SITEMAP_CONCURRENCY`. When processing sitemap index files, each nested sitemap inherits the same limits, but the `MAX_SITEMAP_DEPTH` counter decrements per recursion level, preventing infinite descent into nested indices.

## Code Examples

The following snippet demonstrates invoking the discovery utility that internally respects all six limits:

```typescript
import { discoverSitemapUrls } from "@/server/lib/audit/discovery";

const origin = "https://example.com";
const limit = 100; // Will be clipped to MAX_SITEMAP_DOCS (300) if larger

const { urls, blocked } = await discoverSitemapUrls(origin, { limit });
console.log(`Fetched ${urls.length} URLs from the sitemap`);

```

To inspect a single sitemap document with retry logic and automatic timeout handling:

```typescript
import { fetchSitemapDocumentWithRetry } from "@/server/lib/audit/discovery";

const sitemapUrl = "https://example.com/sitemap.xml";
const result = await fetchSitemapDocumentWithRetry(sitemapUrl);
// Respects SITEMAP_CONCURRENCY, timeout, retries, and size limits internally

```

## Related Files in the Audit Pipeline

- **[`src/server/lib/scrape.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/scrape.ts)** – Consumes the URL array produced by the discovery limits to seed the full site crawl.
- **[`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts)** – Applies complementary workflow‑level memory and time budgets that intersect with sitemap fetch constraints.

## Summary

- **15‑second timeout** aborts stalled sitemap fetches to prevent workflow hangs.
- **Single retry** (`SITEMAP_RETRIES = 1`) offers transient fault tolerance without retry storms.
- **10 MiB size cap** (`MAX_SITEMAP_BYTES`) guards against memory exhaustion from oversized XML.
- **300‑URL ceiling** (`MAX_SITEMAP_DOCS`) keeps batch processing lightweight.
- **5‑request concurrency** (`SITEMAP_CONCURRENCY`) throttles network load.
- **3‑level depth** (`MAX_SITEMAP_DEPTH`) blocks runaway recursion in nested sitemap indexes.

## Frequently Asked Questions

### What happens if a sitemap exceeds the 10 MiB limit?

According to the `every-app/open-seo` source code, any response body that crosses `MAX_SITEMAP_BYTES` triggers an immediate abort, and the sitemap is reported as unreachable in the audit results. The parser never attempts to parse partial payloads.

### Can I increase the 300‑URL limit for large websites?

The constant `MAX_SITEMAP_DOCS` is hard‑coded in [`src/server/lib/audit/discovery.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/discovery.ts) and is not exposed through the public API. To process more than 300 URLs, you must batch requests across multiple sitemap files or fork the repository and rebuild with a higher constant.

### Why does OpenSEO retry only once?

The `SITEMAP_RETRIES = 1` constant balances reliability against audit duration. A single retry catches transient blips (e.g., 502 gateway timeouts) without extending the discovery phase indefinitely. Persistent failures after the second attempt are treated as hard errors.

### How does the depth limit affect sitemap index files?

When the crawler encounters a sitemap index (a sitemap containing `<sitemap>` entries rather than `<url>` entries), it decrements the internal depth counter. If recursion reaches `MAX_SITEMAP_DEPTH`, further nested sitemaps are ignored, preventing exponential URL explosion in deeply nested index structures.