Crawl Budgets and Capacity Constraints in Open SEO Site Audits: A Complete Technical Guide

Open SEO enforces multiple layered budgets—page limits (50–10,000), byte budgets (8 MiB/4 MiB), sliding window concurrency (5–20), and step ceilings (~1,024)—to ensure reliable, cost-controlled site audits within Cloudflare Workers' 128 MiB memory constraints.

The crawl budget and capacity constraints in Open SEO are engineered to balance thoroughness against the hard limits of edge computing. Every audit runs through a distributed crawler implemented in TypeScript, with explicit safeguards in src/server/lib/audit/crawl-window.ts and src/server/workflows/SiteAuditWorkflow.ts preventing memory exhaustion and runaway costs. This guide breaks down each limit, how they interact, and how to work within them.

---

Page Budget: The Primary Crawl Limit

The page budget is the most visible constraint, capping how many URLs an audit will fetch before termination.

Where It's Defined

In src/server/mcp/tools/site-audit-tools.ts, the pageBudget field accepts an integer overriding the default:

// Starting a site audit with explicit page budget
await callMcpTool("run_site_audit", {
  url: "https://example.com",
  pageBudget: 200,           // overrides default: 50 (free) or 10,000 (paid)
  runLighthouse: false,
});

Tier-Based Defaults

Plan	Default	Maximum
Free	50 pages	50 (hard cap)
Paid	10,000 pages	10,000+ (configurable)

Free-plan enforcement happens in src/server/features/audit/services/audit-capacity.ts, which validates the caller's subscription tier before permitting larger values.

---

Byte Budget: Memory Protection for Workers

Cloudflare Workers operate within 128 MiB per isolate. Open SEO's byte budget prevents in-flight HTML from exhausting this limit.

Normal Crawls: 8 MiB Budget

In src/server/lib/audit/crawl-window.ts, the CRAWL_WINDOW constant defines:

const CRAWL_WINDOW = {
  budgetBytes: 8 * 1024 * 1024,  // 8 MiB
  initial: 10,
  min: 5,
  max: 20,
};

This 8 MiB limit applies to UTF-16 HTML held in memory during concurrent fetches. The crawler calculates effective concurrency as:

effectiveWindow = min(maxWindow, floor(budgetBytes / avgPageBytes))

Retry Crawls: 4 MiB Conservative Budget

When a chunk fails mid-crawl—typically from memory pressure—RETRY_CRAWL_WINDOW engages with stricter limits:

const RETRY_CRAWL_WINDOW = {
  budgetBytes: 4 * 1024 * 1024,  // 4 MiB
  initial: 3,
  min: 3,
  max: 10,
};

This halved byte budget and reduced initial window (3 vs. 10) prevent repeated OOM failures on problematic pages.

---

Sliding Window: Dynamic Concurrency Control

The crawl window adjusts fetch concurrency based on real-time performance, keeping audits fast without risking memory exhaustion.

Window Parameters

Parameter	Value	Purpose
initial	10	Starting concurrent fetches
min	5	Floor during degradation
max	20	Ceiling during optimization

Adjustment Logic in adjustCrawlWindow

After each persisted sub-batch, src/server/lib/audit/crawl-window.ts evaluates recent results:

• Slow/error-filled batches: Window halves (minimum min = 5)
• Fast batches (≥ GROWTH_MIN_SAMPLE = 25 pages): Window grows by 5 (maximum max = 20)
• Byte budget binding: Final window never exceeds budgetBytes / avgPageBytes

This adaptive behavior means fast, lightweight sites get crawled more aggressively while slow or heavy pages trigger automatic throttling.

---

Step Budget: The Ultimate Ceiling

Even with generous page and byte budgets, Open SEO's workflow infrastructure imposes a hard step limit of approximately 1,024 steps per audit instance.

Step Consumption Math

In src/server/workflows/SiteAuditWorkflow.ts, the step ledger tracks persisted operations. Each chunk—processing roughly 25 pages—consumes ~3 steps:

maxPages ≈ (1,024 steps / 3 steps per chunk) × 25 pages ≈ 8,000–12,000 pages

This creates an implicit ceiling that can override the explicit pageBudget for very large crawls. The step budget ensures workflow state remains serializable and replayable within Cloudflare's Durable Objects constraints.

---

How Budgets Interact: The Crawl Lifecycle

Understanding the interplay of constraints explains why audits behave as they do:

1. Chunking: src/server/workflows/site-audit-workflow-crawl.ts leases CHUNK_TARGET_PAGES URLs from the audit scratchpad
2. Window leasing: Each chunk respects the current CrawlWindow size, itself bounded by byte budget
3. Adjustment: Post-batch, adjustCrawlWindow recalculates optimal concurrency
4. Retry on failure: OOM or timeout triggers RETRY_CRAWL_WINDOW with 4 MiB budget and initial window of 3
5. Step accounting: Every persisted chunk increments the workflow step ledger; near 1,024 steps, the audit terminates regardless of remaining page budget

---

Free-Plan Constraints: Designed for Cost Control

Free-tier users face stacked limitations ensuring predictable, low-cost resource consumption:

• 50-page hard cap: Enforced at src/server/features/audit/services/audit-capacity.ts
• Standard byte budgets: Same 8 MiB/4 MiB limits as paid tiers
• No priority queueing: Free audits may queue behind paid workloads

Upgrading removes the 50-page ceiling and unlocks the full 10,000-page default with proportional step budget allocation.

---

Monitoring Audit Progress Against Budgets

Poll the audit status to track consumption:

let status;
do {
  status = await callMcpTool("get_audit_status", { auditId });
  console.log(`Phase: ${status.phase}, Pages: ${status.pagesCrawled}/${status.pageBudget}`);
  await new Promise(r => setTimeout(r, 3000));
} while (status.phase !== "finished");

Retrieve crawled page metadata to verify byte-budget compliance:

const pages = await callMcpTool("get_audit_pages", { auditId });
pages.forEach(p => {
  console.log(`${p.url}: ${p.htmlBytes}B HTML, ${p.responseTimeMs}ms`);
});

---

Summary

• Page budget (src/server/mcp/tools/site-audit-tools.ts): 50 pages (free) or 10,000+ (paid), the primary user-visible limit
• Byte budget (src/server/lib/audit/crawl-window.ts): 8 MiB normal, 4 MiB retry—protects against 128 MiB Worker OOM
• Sliding window: 5–20 concurrent fetches, auto-tuned by speed and memory pressure
• Step budget (src/server/workflows/SiteAuditWorkflow.ts): ~1,024 steps caps absolute crawl size at ~8–12k pages
• Free-plan stack: All constraints plus hard 50-page ceiling enforced at audit-capacity.ts

---

Frequently Asked Questions

What happens when a site audit hits its crawl budget?

The audit phase transitions to "finished" with status.budgetExhausted: true. Partial results remain available via get_audit_pages, but the crawler stops fetching new URLs. No automatic extension occurs—you must start a new audit with a higher pageBudget parameter.

Why does my large site audit stop before reaching the page budget?

Check the step budget ceiling. In src/server/workflows/SiteAuditWorkflow.ts, the ~1,024 step limit often binds before pageBudget for sites with many small pages. Each chunk consumes ~3 steps, yielding an effective maximum of 8,000–12,000 pages regardless of your configured budget.

How does Open SEO prevent memory exhaustion on heavy pages?

The byte budget in crawl-window.ts enforces an 8 MiB in-flight HTML limit during normal crawls, dropping to 4 MiB on retry. Combined with dynamic window sizing (effectiveWindow = budgetBytes / avgPageBytes), the system throttles concurrency before approaching Cloudflare's 128 MiB isolate limit.

Can I override the concurrency window for faster crawls?

No direct override exists. The initial, min, and max window values (10, 5, 20) are constants in src/server/lib/audit/crawl-window.ts. However, ensuring your pages are fast and lightweight allows adjustCrawlWindow to automatically grow toward the 20-fetch maximum.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}