How OpenSEO Calculates Crawl Windows for Audits: Budget-Driven Chunking Explained

OpenSEO calculates crawl windows using a dual-budget system that monitors both elapsed time (~90 seconds) and response payload size (~30 MiB) per chunk, dynamically terminating chunks when either resource approaches exhaustion to ensure reliable, resumable site audits.

The crawl window implementation in the every-app/open-seo repository governs how the site audit crawler balances thoroughness against infrastructure constraints. Understanding how OpenSEO calculates crawl windows for audits requires examining the time-byte budget logic implemented in src/server/lib/audit/crawl-window.ts and consumed by the workflow orchestrator.

The Dual-Budget Architecture

OpenSEO enforces crawl windows through two hard caps evaluated during each chunked execution.

Time Budget Constraints

The system defines a soft deadline constant CHUNK_SOFT_DEADLINE_MS, typically set to approximately 90 seconds. This represents the maximum wall-clock time allocated for a single chunk of crawling activity. According to the source code in src/server/lib/audit/crawl-window.ts, the crawler computes remaining time before each fetch operation:

const timeRemaining = deadline - Date.now();

If timeRemaining drops below the safety margin required for the next page fetch, the chunk closes immediately.

Byte Budget Limits

Parallel to time constraints, the crawler tracks raw HTTP response payload volume through the CHUNK_BYTE_BUDGET constant, configured to roughly 30 MiB per chunk. The accumulator bytesFetched increments with every response body received, including error pages and blocked resources. As noted in src/server/lib/audit/crawl-window.ts, the remaining budget calculation follows:

const bytesRemaining = byteBudget - bytesFetched;

This accounting ensures that bandwidth-heavy pages cannot exhaust system resources unpredictably.

Dynamic Window Calculation in Action

The crawl window recalculates continuously throughout the chunk lifecycle. The getCrawlWindow() function exports the current deadline and byteBudget to consuming workflows, allowing real-time adherence checks.

During execution, the workflow in src/server/workflows/siteAuditWorkflowCrawl.ts evaluates both constraints before every fetch:

const { deadline, byteBudget } = getCrawlWindow();

while (urlsToFetch.length && Date.now() < deadline && bytesFetched < byteBudget) {
  const { url } = urlsToFetch.shift()!;
  const { body, size } = await fetchPage(url);
  bytesFetched += size;
  // ...persist page, update frontier
}

When either threshold breaches the minimum safety margin, the loop terminates, the current frontier state persists, and the durable workflow schedules the next chunk.

Chunk Phasing and Frontier Persistence

OpenSEO employs a two-phase approach within each window to optimize resource discovery while respecting limits.

Bootstrap Phase

The first sub-batch of every crawl chunk intentionally restricts itself to 25 pages. This small sample quickly surfaces any early-termination conditions—such as rate limiting or excessive page weights—before committing the full budget.

Target Phase

Subsequent sub-batches pull up to CHUNK_TARGET_PAGES URLs from the scratch-pad data object (DO), constrained by the remaining time and byte allowances. This adaptive sizing prevents the crawler from starting requests it cannot complete within the window.

State Persistence Across Durable Steps

Each chunk persists crawled results to the database via the types defined in src/server/lib/audit/types.ts before the step concludes. This frontier design means the audit resumes after crashes without re-crawling pages, and byte-budget accounting includes failed requests (they still consume transfer overhead).

Orphan Detection and Budget Truncation

The workflow runner executes orphan detection only when the crawl chunk completes naturally. If the crawl window truncates due to CHUNK_SOFT_DEADLINE_MS or CHUNK_BYTE_BUDGET exhaustion, the system skips orphan flagging to prevent false positives on unfinished frontier exploration.

Implementation Reference

The helper crawlPage in src/server/workflows/site-audit-workflow-helpers.ts implements the low-level fetch logic that respects these window boundaries. When initiating audits programmatically, developers can optionally override defaults:

import { startAudit } from "@/server/mcp/tools/site-audit-tools";

await startAudit({
  url: "https://example.com",
  // Optional overrides:
  // crawlTimeMs: 120_000,
  // crawlByteBudget: 50 * 1024 * 1024,
});

Summary

• Dual-budget system: OpenSEO enforces both time (~90s via CHUNK_SOFT_DEADLINE_MS) and byte (~30MiB via CHUNK_BYTE_BUDGET) constraints per chunk.
• Dynamic evaluation: The getCrawlWindow() function provides real-time deadline and byteBudget values checked before every fetch.
• Phased execution: A 25-page bootstrap phase precedes the full CHUNK_TARGET_PAGES pull to validate site behavior early.
• Durable state: Frontier persistence in src/server/lib/audit/types.ts ensures crawls resume correctly after interruptions.
• Truncation awareness: Orphan detection runs only on naturally completed chunks, avoiding false flags when budgets exhaust.

Frequently Asked Questions

What happens if a single page exceeds the byte budget?

If an individual HTTP response exceeds the remaining CHUNK_BYTE_BUDGET, the crawler records the page and closes the chunk. The byte accounting includes the actual transferred size even for partial or failed downloads, ensuring accurate budget tracking in src/server/lib/audit/crawl-window.ts.

Can crawl window limits be customized per audit?

Yes. When calling startAudit() from @/server/mcp/tools/site-audit-tools, you can pass crawlTimeMs to override CHUNK_SOFT_DEADLINE_MS and crawlByteBudget to override CHUNK_BYTE_BUDGET. These parameters propagate to getCrawlWindow() for that specific audit execution.

Why does OpenSEO use a soft deadline instead of a hard timeout?

The soft deadline (CHUNK_SOFT_DEADLINE_MS) allows the crawler to finish the current in-flight request rather than aborting mid-stream. This prevents corrupting the frontier state in src/server/workflows/siteAuditWorkflowCrawl.ts and ensures the durable workflow can cleanly persist progress before resuming.

How does the crawler handle sites with thousands of pages?

The chunking architecture in every-app/open-seo breaks large crawls into multiple durable steps. Each chunk processes up to its budget limits, persists the frontier via src/server/lib/audit/types.ts, and queues the next workflow step. This pattern prevents memory exhaustion and allows multi-hour audits to complete across distributed execution slices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}