# How OpenSEO Handles Large Website Audits: Scalable Cloudflare Workers Architecture

> Discover how OpenSEO handles large website audits with its scalable Cloudflare Workers architecture. Learn about partitioning, durable scratchpads, and adaptive concurrency for efficient processing.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: architecture
- Published: 2026-08-21

---

**OpenSEO processes websites with tens of thousands of pages by partitioning audits into bounded Discovery, Crawl, and Lighthouse phases, utilizing a durable scratchpad frontier, chunked execution, and adaptive concurrency to operate within Cloudflare Workers’ strict memory and step-output constraints.**

OpenSEO is engineered to audit enterprise-scale sites without exceeding serverless platform limits. The architecture treats large-scale crawling as a resource-constrained workflow, implementing hard ceilings on page counts, batch sizes, and in-flight memory to guarantee predictable performance across tens of thousands of URLs.

## Enforcing Hard Page Limits with Tiered Caps

The system prevents resource exhaustion by enforcing **bounded page limits** before execution begins. In [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts), the audit request schema defines `maxPages` with tier-specific hard caps: `FREE_MAX_AUDIT_PAGES` for free users and `PAID_MAX_AUDIT_PAGES` for paid plans.

The `AuditService` class in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) clamps user-provided values to these ceilings during initialization. This ensures the workflow never attempts to process an unbounded URL set, protecting both system stability and billing predictability.

## Discovery Phase: Seeding the Durable Scratchpad

To avoid exceeding Cloudflare’s ~1 MiB step-output limit, the **Discovery phase** never returns massive URL lists directly. Instead, `discoverUrls` in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) extracts URLs from the start domain and sitemaps, then writes **only seed URLs** into a durable **audit scratchpad** implemented as a Cloudflare Durable Object (DO).

Seeding occurs in batches of `SEED_RPC_BATCH = 2,000` to keep RPC payloads small and durable object transactions efficient. This approach decouples URL enumeration from the workflow state, allowing the system to handle discovery results that would otherwise exceed memory limits.

## Crawl Phase: Chunked Execution with Soft Deadlines

The **Crawl phase** processes URLs in discrete chunks to maintain predictable resource usage. As implemented in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), each execution claims up to `CHUNK_TARGET_PAGES = 200` URLs from the scratchpad per chunk, respecting the global `maxPages` limit enforced during initialization.

A **soft deadline** mechanism (`CHUNK_SOFT_DEADLINE_MS = 90,000` ms) halts new fetches after approximately 90 seconds, preventing worker timeouts on slow sites. Pages persist in **sub-batches** of `PERSIST_BATCH_SIZE = 25`, pipelining database writes sequentially to cap memory pressure rather than buffering entire chunks in RAM.

### Limiting Link Discovery Scope

To prevent unbounded memory growth from link extraction, the crawler implements dual ceilings:

- **Stored links per page** are capped at `MAX_STORED_LINKS_PER_PAGE = 500`
- **Discovered URLs per batch** are limited to `MAX_DISCOVERED_PER_BATCH = 20,000`

These boundaries in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) ensure that pages with excessive navigation links or crawl traps cannot exhaust available memory or storage quotas.

## Adaptive Concurrency and Memory Budgeting

OpenSEO optimizes throughput using an **adaptive concurrency window** defined in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts). The system initializes with `INITIAL_CRAWL_WINDOW = 10` concurrent fetches and dynamically adjusts based on recent batch performance.

The `adjustCrawlWindow` function evaluates fetch success rates and response times, shrinking the window when encountering slow or error-prone pages and expanding it for fast responses. Crucially, the window respects an **in-flight HTML budget** of `IN_FLIGHT_HTML_BUDGET_BYTES = 16 MiB`, ensuring total buffered response bodies never exceed platform memory constraints.

## Fault Tolerance via Progress Tracking

The architecture supports safe step replay through durable progress counters. Each crawl chunk reports `attempted` and `pending` metrics to `AuditProgressKV`, as implemented in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts).

If a chunk reports zero progress, the workflow triggers an early exit to prevent infinite loops on stuck domains. This stateless progress tracking allows the workflow to resume mid-audit without reprocessing completed chunks, essential for large sites requiring multiple worker invocations.

## Selective Lighthouse Analysis

For the final **Lighthouse phase**, OpenSEO avoids comprehensive performance scans that would overwhelm large sites. As configured in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), the `LIGHTHOUSE_FETCH_STEP` executes only on a representative sample of crawled pages rather than the full `maxPages` set. This sampling strategy delivers actionable Core Web Vitals data without proportional resource scaling.

## Launching a Bounded Audit

To initiate an audit with custom limits:

```typescript
import { AuditService } from "@/server/features/audit/services/AuditService";

export async function startAudit(
  projectId: string,
  startUrl: string,
  maxPages?: number,
) {
  // maxPages is automatically clamped to plan limits
  const auditId = await AuditService.start({
    projectId,
    startUrl,
    maxPages,
    lighthouseStrategy: "auto",
  });
  return auditId;
}

```

The `AuditService.start` method validates and clamps the `maxPages` parameter against tier limits before workflow invocation.

## Summary

- **Hard limits**: The system enforces tier-specific `maxPages` caps in [`AuditService.ts`](https://github.com/every-app/open-seo/blob/main/AuditService.ts) before workflow execution begins.
- **Scratchpad architecture**: Discovery seeds URLs into a Cloudflare DO in 2,000-item batches, avoiding step-output size limits.
- **Chunked crawling**: The crawl phase processes 200-page chunks with 90-second soft deadlines and 25-item persistence batches.
- **Memory safety**: Link discovery is capped at 500 links per page and 20,000 discovered URLs per batch.
- **Adaptive throughput**: The rolling concurrency window adjusts between 10 and maximum safe concurrency while respecting a 16 MiB in-flight HTML budget.
- **Fault tolerance**: Progress tracking via KV store enables safe replays and early exit on stalled chunks.

## Frequently Asked Questions

### What is the maximum number of pages OpenSEO can audit in a single run?

The absolute maximum depends on the subscription tier, defined by `PAID_MAX_AUDIT_PAGES` and `FREE_MAX_AUDIT_PAGES` in [`src/server/lib/audit/types.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/types.ts). The `AuditService` clamps any user-provided `maxPages` value to these hard ceilings before the workflow initiates, ensuring the system never attempts to process more pages than the infrastructure supports.

### How does OpenSEO prevent hitting Cloudflare Workers' step output limits?

Instead of returning discovered URLs in step outputs, the **Discovery phase** writes seed URLs directly to a durable scratchpad Durable Object. Batching seeds in groups of `SEED_RPC_BATCH = 2,000` keeps individual RPC payloads well below the ~1 MiB step-output threshold, while the scratchpad frontier maintains state across workflow steps.

### What happens if a website has pages with thousands of internal links?

The crawler implements safeguards in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) by capping stored links at `MAX_STORED_LINKS_PER_PAGE = 500` and discovered URLs per batch at `MAX_DISCOVERED_PER_BATCH = 20,000`. These limits prevent individual pages or crawl traps from consuming excessive memory or storage resources.

### How does the system adapt to slow or unresponsive web servers?

The `adjustCrawlWindow` function in [`src/server/lib/audit/crawl-window.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/crawl-window.ts) monitors recent batch performance and automatically shrinks the concurrency window when encountering slow responses or errors. This adaptive behavior, combined with the `CHUNK_SOFT_DEADLINE_MS = 90,000` ms deadline per chunk, ensures workers complete within platform limits even when targeting unreliable infrastructure.