# How to Configure Site Audit Crawl Window Limits and Page Budgets in Open SEO

> Learn to configure site audit crawl window limits and page budgets in Open SEO. Control total pages crawled with page_budget and memory usage via OPEN_SEO_CRAWL_WINDOW_BYTES.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-08-10

---

**Set `page_budget` in your API call to limit total pages crawled, and use the `OPEN_SEO_CRAWL_WINDOW_BYTES` environment variable to control per-chunk memory usage.**

Open SEO's distributed site-audit crawler processes targets in manageable chunks controlled by two key parameters: **page budget** and **crawl window byte budget**. This guide explains how to configure both limits based on the actual implementation in the every-app/open-seo repository.

## Understanding the Two Budget Controls

Open SEO applies limits at different layers of the crawl pipeline:

- **Page budget** – Hard cap on total URLs fetched across all chunks
- **Crawl window byte budget** – Soft cap on HTTP response size per individual chunk

These work together to prevent runaway resource consumption while ensuring crawl progress is recoverable.

## Configuring the Page Budget

The **page budget** is the primary control most users need. It sets the maximum number of pages the audit will attempt to crawl before stopping, regardless of remaining URLs in the frontier.

### Default and Limits

| Tier | Default | Maximum |
|------|---------|---------|
| Free | 50 pages | 50 pages |
| Paid | Configurable | Higher limits apply |

### API Configuration

Pass `page_budget` in your **Site Audit** endpoint request. The field is validated by the Zod schema defined at line 55 of [[`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts)](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts#L55):

```typescript
// Start audit with custom 200-page limit
const response = await fetch('/api/mcp/site_audit/start', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    url: 'https://example.com',
    page_budget: 200  // ← custom limit here
  })
});

const { auditId } = await response.json();

```

The crawler stops immediately when `pagesCrawled` reaches this threshold. Partial chunks are persisted, and the audit proceeds to subsequent phases (link checks, Lighthouse sampling).

## Adjusting the Crawl Window Byte Budget

The **crawl window byte budget** protects individual worker instances from memory pressure and RPC size limits. It controls how much response data a single chunk may download before yielding.

### Default Behavior

- Default: approximately **32 MiB** per chunk
- Applied per-chunk, not globally
- When exhausted, current chunk finishes early; remaining URLs deferred to next chunk

### Self-Hosted Configuration

Override via environment variable in [[`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts):

```bash

# .env file for self-hosted instances

OPEN_SEO_CRAWL_WINDOW_BYTES=16777216  # 16 MiB instead of 32 MiB

```

The `CrawlWindow` constructor reads this variable and initializes counters tracked in the `CrawlWindow` type ([[`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts)):

- `pagesCrawled` – incremented per successful fetch
- `bytesFetched` – summed from `Content-Length` or actual response size

## How Limits Are Enforced During Crawls

The crawl pipeline applies budgets through four stages:

1. **API validation** – `page_budget` validated by Zod schema, stored in audit record
2. **Window initialization** – Each chunk creates a fresh `CrawlWindow` with counters at zero
3. **Runtime checking** – After each fetch, counters checked against limits
4. **Early termination** – Chunk persists progress and exits if either budget exceeded

Each chunk also has a **soft time budget** (~90 seconds) with headroom for persistence operations.

## Monitoring Crawl Progress and Budget Status

Track real-time consumption via the progress KV store ([[`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts)):

```typescript
// Poll for crawl status and budget indicators
const status = await fetch(
  `/api/mcp/site_audit/status?auditId=${auditId}`
).then(r => r.json());

console.log({
  pagesCrawled: status.pages_crawled,
  pagesTotal: status.pages_total,
  crawlCompleted: status.crawl_completed,
  stoppedEarly: !status.crawl_completed && status.pages_crawled >= status.pages_total
});

```

Key fields exposed:
- `pages_crawled` – current count against your `page_budget`
- `pages_total` – ceiling from configured budget
- `crawl_completed` – `false` indicates truncation due to budget limits

## Practical Configuration Patterns

### Small Sites, Tight Budgets

```typescript
// Fast audit for landing page analysis
{ page_budget: 10 }

```

### Large Sites, Chunked Approach

```bash

# Reduce per-chunk memory for constrained workers

OPEN_SEO_CRAWL_WINDOW_BYTES=8388608  # 8 MiB

```

```typescript
// Higher page count with smaller chunks
{ page_budget: 1000 }

```

### Validation Before Full Crawl

```typescript
// Quick validation crawl followed by full audit if healthy
const validation = await startAudit({ url, page_budget: 5 });
if (validation.severity !== 'critical') {
  await startAudit({ url, page_budget: 500 });
}

```

## Summary

- **Page budget** controls total crawl scope via `page_budget` field in API calls (default: 50, schema at [`site-audit-tools.ts:55`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts#L55))
- **Byte budget** controls per-chunk memory via `OPEN_SEO_CRAWL_WINDOW_BYTES` environment variable (default: ~32 MiB, implementation in [[`crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/crawl-window.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts))
- Both limits trigger **early chunk termination** with progress persisted for continuation
- Monitor consumption through [[`progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/progress-kv.ts)](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts) status endpoint

## Frequently Asked Questions

### What happens when the page budget is reached mid-crawl?

The crawler stops issuing new fetches immediately. Already-fetched pages in the current chunk are processed normally, then the audit advances to post-crawl phases (link validation, Lighthouse checks). The UI shows `crawl_completed: false` to indicate truncation.

### Can I change budgets for a running audit?

No. Both `page_budget` and crawl window configuration are **immutable per audit**. Create a new audit with adjusted parameters if you need different limits.

### Why does my crawl stop before the page budget is exhausted?

The **byte budget** likely triggered early chunk termination. Large pages, heavy media responses, or slow compression can inflate `bytesFetched` faster than `pagesCrawled`. Reduce `OPEN_SEO_CRAWL_WINDOW_BYTES` to force smaller chunks, or increase it if workers have ample memory.

### How do I estimate appropriate byte budget settings?

Target roughly **5-10 pages per MiB** for typical content sites. A 32 MiB window handles ~160-320 average pages. News sites or media-heavy pages may need 1-2 MiB each—adjust downward accordingly.