How to Configure Site Audit Crawl Window Limits and Page Budgets in Open SEO

Set page_budget in your API call to limit total pages crawled, and use the OPEN_SEO_CRAWL_WINDOW_BYTES environment variable to control per-chunk memory usage.

Open SEO's distributed site-audit crawler processes targets in manageable chunks controlled by two key parameters: page budget and crawl window byte budget. This guide explains how to configure both limits based on the actual implementation in the every-app/open-seo repository.

Understanding the Two Budget Controls

Open SEO applies limits at different layers of the crawl pipeline:

• Page budget – Hard cap on total URLs fetched across all chunks
• Crawl window byte budget – Soft cap on HTTP response size per individual chunk

These work together to prevent runaway resource consumption while ensuring crawl progress is recoverable.

Configuring the Page Budget

The page budget is the primary control most users need. It sets the maximum number of pages the audit will attempt to crawl before stopping, regardless of remaining URLs in the frontier.

Default and Limits

Tier	Default	Maximum
Free	50 pages	50 pages
Paid	Configurable	Higher limits apply

API Configuration

Pass page_budget in your Site Audit endpoint request. The field is validated by the Zod schema defined at line 55 of [src/server/mcp/tools/site-audit-tools.ts](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts#L55):

// Start audit with custom 200-page limit
const response = await fetch('/api/mcp/site_audit/start', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    url: 'https://example.com',
    page_budget: 200  // ← custom limit here
  })
});

const { auditId } = await response.json();

The crawler stops immediately when pagesCrawled reaches this threshold. Partial chunks are persisted, and the audit proceeds to subsequent phases (link checks, Lighthouse sampling).

Adjusting the Crawl Window Byte Budget

The crawl window byte budget protects individual worker instances from memory pressure and RPC size limits. It controls how much response data a single chunk may download before yielding.

Default Behavior

• Default: approximately 32 MiB per chunk
• Applied per-chunk, not globally
• When exhausted, current chunk finishes early; remaining URLs deferred to next chunk

Self-Hosted Configuration

Override via environment variable in [src/server/lib/audit/crawl-window.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts):

# .env file for self-hosted instances

OPEN_SEO_CRAWL_WINDOW_BYTES=16777216  # 16 MiB instead of 32 MiB

The CrawlWindow constructor reads this variable and initializes counters tracked in the CrawlWindow type ([src/server/lib/audit/types.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts)):

• pagesCrawled – incremented per successful fetch
• bytesFetched – summed from Content-Length or actual response size

How Limits Are Enforced During Crawls

The crawl pipeline applies budgets through four stages:

1. API validation – page_budget validated by Zod schema, stored in audit record
2. Window initialization – Each chunk creates a fresh CrawlWindow with counters at zero
3. Runtime checking – After each fetch, counters checked against limits
4. Early termination – Chunk persists progress and exits if either budget exceeded

Each chunk also has a soft time budget (~90 seconds) with headroom for persistence operations.

Monitoring Crawl Progress and Budget Status

Track real-time consumption via the progress KV store ([src/server/lib/audit/progress-kv.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)):

// Poll for crawl status and budget indicators
const status = await fetch(
  `/api/mcp/site_audit/status?auditId=${auditId}`
).then(r => r.json());

console.log({
  pagesCrawled: status.pages_crawled,
  pagesTotal: status.pages_total,
  crawlCompleted: status.crawl_completed,
  stoppedEarly: !status.crawl_completed && status.pages_crawled >= status.pages_total
});

Key fields exposed:

• pages_crawled – current count against your page_budget
• pages_total – ceiling from configured budget
• crawl_completed – false indicates truncation due to budget limits

Practical Configuration Patterns

Small Sites, Tight Budgets

// Fast audit for landing page analysis
{ page_budget: 10 }

Large Sites, Chunked Approach

# Reduce per-chunk memory for constrained workers

OPEN_SEO_CRAWL_WINDOW_BYTES=8388608  # 8 MiB

// Higher page count with smaller chunks
{ page_budget: 1000 }

Validation Before Full Crawl

// Quick validation crawl followed by full audit if healthy
const validation = await startAudit({ url, page_budget: 5 });
if (validation.severity !== 'critical') {
  await startAudit({ url, page_budget: 500 });
}

Summary

• Page budget controls total crawl scope via page_budget field in API calls (default: 50, schema at site-audit-tools.ts:55)
• Byte budget controls per-chunk memory via OPEN_SEO_CRAWL_WINDOW_BYTES environment variable (default: ~32 MiB, implementation in [crawl-window.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts))
• Both limits trigger early chunk termination with progress persisted for continuation
• Monitor consumption through [progress-kv.ts](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) status endpoint

Frequently Asked Questions

What happens when the page budget is reached mid-crawl?

The crawler stops issuing new fetches immediately. Already-fetched pages in the current chunk are processed normally, then the audit advances to post-crawl phases (link validation, Lighthouse checks). The UI shows crawl_completed: false to indicate truncation.

Can I change budgets for a running audit?

No. Both page_budget and crawl window configuration are immutable per audit. Create a new audit with adjusted parameters if you need different limits.

Why does my crawl stop before the page budget is exhausted?

The byte budget likely triggered early chunk termination. Large pages, heavy media responses, or slow compression can inflate bytesFetched faster than pagesCrawled. Reduce OPEN_SEO_CRAWL_WINDOW_BYTES to force smaller chunks, or increase it if workers have ample memory.

How do I estimate appropriate byte budget settings?

Target roughly 5-10 pages per MiB for typical content sites. A 32 MiB window handles ~160-320 average pages. News sites or media-heavy pages may need 1-2 MiB each—adjust downward accordingly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}