# How OpenSEO Calculates Crawl Windows for Audits: Budget-Driven Chunking Explained

> Discover how OpenSEO calculates crawl windows using a dual budget system for time and payload size ensuring reliable and resumable site audits.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**OpenSEO calculates crawl windows using a dual-budget system that monitors both elapsed time (~90 seconds) and response payload size (~30 MiB) per chunk, dynamically terminating chunks when either resource approaches exhaustion to ensure reliable, resumable site audits.**

The crawl window implementation in the `every-app/open-seo` repository governs how the site audit crawler balances thoroughness against infrastructure constraints. Understanding how OpenSEO calculates crawl windows for audits requires examining the time-byte budget logic implemented in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts) and consumed by the workflow orchestrator.

## The Dual-Budget Architecture

OpenSEO enforces crawl windows through two hard caps evaluated during each chunked execution.

### Time Budget Constraints

The system defines a soft deadline constant `CHUNK_SOFT_DEADLINE_MS`, typically set to approximately 90 seconds. This represents the maximum wall-clock time allocated for a single chunk of crawling activity. According to the source code in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts), the crawler computes remaining time before each fetch operation:

```typescript
const timeRemaining = deadline - Date.now();

```

If `timeRemaining` drops below the safety margin required for the next page fetch, the chunk closes immediately.

### Byte Budget Limits

Parallel to time constraints, the crawler tracks raw HTTP response payload volume through the `CHUNK_BYTE_BUDGET` constant, configured to roughly 30 MiB per chunk. The accumulator `bytesFetched` increments with every response body received, including error pages and blocked resources. As noted in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts), the remaining budget calculation follows:

```typescript
const bytesRemaining = byteBudget - bytesFetched;

```

This accounting ensures that bandwidth-heavy pages cannot exhaust system resources unpredictably.

## Dynamic Window Calculation in Action

The crawl window recalculates continuously throughout the chunk lifecycle. The `getCrawlWindow()` function exports the current `deadline` and `byteBudget` to consuming workflows, allowing real-time adherence checks.

During execution, the workflow in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) evaluates both constraints before every fetch:

```typescript
const { deadline, byteBudget } = getCrawlWindow();

while (urlsToFetch.length && Date.now() < deadline && bytesFetched < byteBudget) {
  const { url } = urlsToFetch.shift()!;
  const { body, size } = await fetchPage(url);
  bytesFetched += size;
  // ...persist page, update frontier
}

```

When either threshold breaches the minimum safety margin, the loop terminates, the current frontier state persists, and the durable workflow schedules the next chunk.

## Chunk Phasing and Frontier Persistence

OpenSEO employs a two-phase approach within each window to optimize resource discovery while respecting limits.

### Bootstrap Phase

The first sub-batch of every crawl chunk intentionally restricts itself to 25 pages. This small sample quickly surfaces any early-termination conditions—such as rate limiting or excessive page weights—before committing the full budget.

### Target Phase

Subsequent sub-batches pull up to `CHUNK_TARGET_PAGES` URLs from the scratch-pad data object (DO), constrained by the remaining time and byte allowances. This adaptive sizing prevents the crawler from starting requests it cannot complete within the window.

### State Persistence Across Durable Steps

Each chunk persists crawled results to the database via the types defined in [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts) before the step concludes. This frontier design means the audit resumes after crashes without re-crawling pages, and byte-budget accounting includes failed requests (they still consume transfer overhead).

## Orphan Detection and Budget Truncation

The workflow runner executes orphan detection only when the crawl chunk completes naturally. If the crawl window truncates due to `CHUNK_SOFT_DEADLINE_MS` or `CHUNK_BYTE_BUDGET` exhaustion, the system skips orphan flagging to prevent false positives on unfinished frontier exploration.

## Implementation Reference

The helper `crawlPage` in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) implements the low-level fetch logic that respects these window boundaries. When initiating audits programmatically, developers can optionally override defaults:

```typescript
import { startAudit } from "@/server/mcp/tools/site-audit-tools";

await startAudit({
  url: "https://example.com",
  // Optional overrides:
  // crawlTimeMs: 120_000,
  // crawlByteBudget: 50 * 1024 * 1024,
});

```

## Summary

- **Dual-budget system**: OpenSEO enforces both time (~90s via `CHUNK_SOFT_DEADLINE_MS`) and byte (~30MiB via `CHUNK_BYTE_BUDGET`) constraints per chunk.
- **Dynamic evaluation**: The `getCrawlWindow()` function provides real-time `deadline` and `byteBudget` values checked before every fetch.
- **Phased execution**: A 25-page bootstrap phase precedes the full `CHUNK_TARGET_PAGES` pull to validate site behavior early.
- **Durable state**: Frontier persistence in [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts) ensures crawls resume correctly after interruptions.
- **Truncation awareness**: Orphan detection runs only on naturally completed chunks, avoiding false flags when budgets exhaust.

## Frequently Asked Questions

### What happens if a single page exceeds the byte budget?

If an individual HTTP response exceeds the remaining `CHUNK_BYTE_BUDGET`, the crawler records the page and closes the chunk. The byte accounting includes the actual transferred size even for partial or failed downloads, ensuring accurate budget tracking in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts).

### Can crawl window limits be customized per audit?

Yes. When calling `startAudit()` from `@/server/mcp/tools/site-audit-tools`, you can pass `crawlTimeMs` to override `CHUNK_SOFT_DEADLINE_MS` and `crawlByteBudget` to override `CHUNK_BYTE_BUDGET`. These parameters propagate to `getCrawlWindow()` for that specific audit execution.

### Why does OpenSEO use a soft deadline instead of a hard timeout?

The soft deadline (`CHUNK_SOFT_DEADLINE_MS`) allows the crawler to finish the current in-flight request rather than aborting mid-stream. This prevents corrupting the frontier state in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) and ensures the durable workflow can cleanly persist progress before resuming.

### How does the crawler handle sites with thousands of pages?

The chunking architecture in `every-app/open-seo` breaks large crawls into multiple durable steps. Each chunk processes up to its budget limits, persists the frontier via [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts), and queues the next workflow step. This pattern prevents memory exhaustion and allows multi-hour audits to complete across distributed execution slices.