# Crawl Budgets and Capacity Constraints in Open SEO Site Audits: A Complete Technical Guide

> Master crawl budgets & capacity constraints in OpenSEO site audits. Learn how layered budgets ensure reliable, cost-controlled audits within Cloudflare Workers' limits. Optimize your technical SEO today.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**Open SEO enforces multiple layered budgets—page limits (50–10,000), byte budgets (8 MiB/4 MiB), sliding window concurrency (5–20), and step ceilings (~1,024)—to ensure reliable, cost-controlled site audits within Cloudflare Workers' 128 MiB memory constraints.**

The **crawl budget** and **capacity constraints** in Open SEO are engineered to balance thoroughness against the hard limits of edge computing. Every audit runs through a distributed crawler implemented in TypeScript, with explicit safeguards in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts) and [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) preventing memory exhaustion and runaway costs. This guide breaks down each limit, how they interact, and how to work within them.

---

## Page Budget: The Primary Crawl Limit

The **page budget** is the most visible constraint, capping how many URLs an audit will fetch before termination.

### Where It's Defined

In [`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts), the `pageBudget` field accepts an integer overriding the default:

```ts
// Starting a site audit with explicit page budget
await callMcpTool("run_site_audit", {
  url: "https://example.com",
  pageBudget: 200,           // overrides default: 50 (free) or 10,000 (paid)
  runLighthouse: false,
});

```

### Tier-Based Defaults

| Plan | Default | Maximum |
|------|---------|---------|
| Free | **50 pages** | 50 (hard cap) |
| Paid | 10,000 pages | 10,000+ (configurable) |

Free-plan enforcement happens in [`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts), which validates the caller's subscription tier before permitting larger values.

---

## Byte Budget: Memory Protection for Workers

Cloudflare Workers operate within **128 MiB per isolate**. Open SEO's **byte budget** prevents in-flight HTML from exhausting this limit.

### Normal Crawls: 8 MiB Budget

In [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts), the `CRAWL_WINDOW` constant defines:

```ts
const CRAWL_WINDOW = {
  budgetBytes: 8 * 1024 * 1024,  // 8 MiB
  initial: 10,
  min: 5,
  max: 20,
};

```

This **8 MiB** limit applies to UTF-16 HTML held in memory during concurrent fetches. The crawler calculates effective concurrency as:

```

effectiveWindow = min(maxWindow, floor(budgetBytes / avgPageBytes))

```

### Retry Crawls: 4 MiB Conservative Budget

When a chunk fails mid-crawl—typically from memory pressure—`RETRY_CRAWL_WINDOW` engages with stricter limits:

```ts
const RETRY_CRAWL_WINDOW = {
  budgetBytes: 4 * 1024 * 1024,  // 4 MiB
  initial: 3,
  min: 3,
  max: 10,
};

```

This **halved byte budget** and reduced initial window (3 vs. 10) prevent repeated OOM failures on problematic pages.

---

## Sliding Window: Dynamic Concurrency Control

The **crawl window** adjusts fetch concurrency based on real-time performance, keeping audits fast without risking memory exhaustion.

### Window Parameters

| Parameter | Value | Purpose |
|-----------|-------|---------|
| `initial` | 10 | Starting concurrent fetches |
| `min` | 5 | Floor during degradation |
| `max` | 20 | Ceiling during optimization |

### Adjustment Logic in `adjustCrawlWindow`

After each persisted sub-batch, [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts) evaluates recent results:

- **Slow/error-filled batches**: Window halves (minimum `min` = 5)
- **Fast batches** (≥ `GROWTH_MIN_SAMPLE` = 25 pages): Window grows by 5 (maximum `max` = 20)
- **Byte budget binding**: Final window never exceeds `budgetBytes / avgPageBytes`

This adaptive behavior means **fast, lightweight sites get crawled more aggressively** while **slow or heavy pages trigger automatic throttling**.

---

## Step Budget: The Ultimate Ceiling

Even with generous page and byte budgets, Open SEO's workflow infrastructure imposes a hard **step limit** of approximately **1,024 steps per audit instance**.

### Step Consumption Math

In [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts), the step ledger tracks persisted operations. Each chunk—processing roughly 25 pages—consumes ~3 steps:

```

maxPages ≈ (1,024 steps / 3 steps per chunk) × 25 pages ≈ 8,000–12,000 pages

```

This creates an implicit ceiling that can **override the explicit `pageBudget`** for very large crawls. The step budget ensures workflow state remains serializable and replayable within Cloudflare's Durable Objects constraints.

---

## How Budgets Interact: The Crawl Lifecycle

Understanding the interplay of constraints explains why audits behave as they do:

1. **Chunking**: [`src/server/workflows/site-audit-workflow-crawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-crawl.ts) leases `CHUNK_TARGET_PAGES` URLs from the audit scratchpad
2. **Window leasing**: Each chunk respects the current `CrawlWindow` size, itself bounded by byte budget
3. **Adjustment**: Post-batch, `adjustCrawlWindow` recalculates optimal concurrency
4. **Retry on failure**: OOM or timeout triggers `RETRY_CRAWL_WINDOW` with 4 MiB budget and initial window of 3
5. **Step accounting**: Every persisted chunk increments the workflow step ledger; near 1,024 steps, the audit terminates regardless of remaining page budget

---

## Free-Plan Constraints: Designed for Cost Control

Free-tier users face stacked limitations ensuring **predictable, low-cost resource consumption**:

- **50-page hard cap**: Enforced at [`src/server/features/audit/services/audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/audit-capacity.ts)
- **Standard byte budgets**: Same 8 MiB/4 MiB limits as paid tiers
- **No priority queueing**: Free audits may queue behind paid workloads

Upgrading removes the 50-page ceiling and unlocks the full 10,000-page default with proportional step budget allocation.

---

## Monitoring Audit Progress Against Budgets

Poll the audit status to track consumption:

```ts
let status;
do {
  status = await callMcpTool("get_audit_status", { auditId });
  console.log(`Phase: ${status.phase}, Pages: ${status.pagesCrawled}/${status.pageBudget}`);
  await new Promise(r => setTimeout(r, 3000));
} while (status.phase !== "finished");

```

Retrieve crawled page metadata to verify byte-budget compliance:

```ts
const pages = await callMcpTool("get_audit_pages", { auditId });
pages.forEach(p => {
  console.log(`${p.url}: ${p.htmlBytes}B HTML, ${p.responseTimeMs}ms`);
});

```

---

## Summary

- **Page budget** ([`src/server/mcp/tools/site-audit-tools.ts`](https://github.com/every-app/open-seo/blob/main/src/server/mcp/tools/site-audit-tools.ts)): 50 pages (free) or 10,000+ (paid), the primary user-visible limit
- **Byte budget** ([`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts)): 8 MiB normal, 4 MiB retry—protects against 128 MiB Worker OOM
- **Sliding window**: 5–20 concurrent fetches, auto-tuned by speed and memory pressure
- **Step budget** ([`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts)): ~1,024 steps caps absolute crawl size at ~8–12k pages
- **Free-plan stack**: All constraints plus hard 50-page ceiling enforced at [`audit-capacity.ts`](https://github.com/every-app/open-seo/blob/main/audit-capacity.ts)

---

## Frequently Asked Questions

### What happens when a site audit hits its crawl budget?

The audit phase transitions to **"finished"** with `status.budgetExhausted: true`. Partial results remain available via `get_audit_pages`, but the crawler stops fetching new URLs. No automatic extension occurs—you must start a new audit with a higher `pageBudget` parameter.

### Why does my large site audit stop before reaching the page budget?

Check the **step budget** ceiling. In [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts), the ~1,024 step limit often binds before `pageBudget` for sites with many small pages. Each chunk consumes ~3 steps, yielding an effective maximum of 8,000–12,000 pages regardless of your configured budget.

### How does Open SEO prevent memory exhaustion on heavy pages?

The **byte budget** in [`crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/crawl-window.ts) enforces an 8 MiB in-flight HTML limit during normal crawls, dropping to 4 MiB on retry. Combined with dynamic window sizing (`effectiveWindow = budgetBytes / avgPageBytes`), the system throttles concurrency before approaching Cloudflare's 128 MiB isolate limit.

### Can I override the concurrency window for faster crawls?

No direct override exists. The `initial`, `min`, and `max` window values (10, 5, 20) are constants in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts). However, ensuring your pages are fast and lightweight allows `adjustCrawlWindow` to automatically grow toward the 20-fetch maximum.