How OpenSEO Handles Large Website Audits: Scalable Cloudflare Workers Architecture

OpenSEO processes websites with tens of thousands of pages by partitioning audits into bounded Discovery, Crawl, and Lighthouse phases, utilizing a durable scratchpad frontier, chunked execution, and adaptive concurrency to operate within Cloudflare Workers’ strict memory and step-output constraints.

OpenSEO is engineered to audit enterprise-scale sites without exceeding serverless platform limits. The architecture treats large-scale crawling as a resource-constrained workflow, implementing hard ceilings on page counts, batch sizes, and in-flight memory to guarantee predictable performance across tens of thousands of URLs.

Enforcing Hard Page Limits with Tiered Caps

The system prevents resource exhaustion by enforcing bounded page limits before execution begins. In src/server/lib/audit/types.ts, the audit request schema defines maxPages with tier-specific hard caps: FREE_MAX_AUDIT_PAGES for free users and PAID_MAX_AUDIT_PAGES for paid plans.

The AuditService class in src/server/features/audit/services/AuditService.ts clamps user-provided values to these ceilings during initialization. This ensures the workflow never attempts to process an unbounded URL set, protecting both system stability and billing predictability.

Discovery Phase: Seeding the Durable Scratchpad

To avoid exceeding Cloudflare’s ~1 MiB step-output limit, the Discovery phase never returns massive URL lists directly. Instead, discoverUrls in src/server/workflows/siteAuditWorkflowPhases.ts extracts URLs from the start domain and sitemaps, then writes only seed URLs into a durable audit scratchpad implemented as a Cloudflare Durable Object (DO).

Seeding occurs in batches of SEED_RPC_BATCH = 2,000 to keep RPC payloads small and durable object transactions efficient. This approach decouples URL enumeration from the workflow state, allowing the system to handle discovery results that would otherwise exceed memory limits.

Crawl Phase: Chunked Execution with Soft Deadlines

The Crawl phase processes URLs in discrete chunks to maintain predictable resource usage. As implemented in src/server/workflows/siteAuditWorkflowCrawl.ts, each execution claims up to CHUNK_TARGET_PAGES = 200 URLs from the scratchpad per chunk, respecting the global maxPages limit enforced during initialization.

A soft deadline mechanism (CHUNK_SOFT_DEADLINE_MS = 90,000 ms) halts new fetches after approximately 90 seconds, preventing worker timeouts on slow sites. Pages persist in sub-batches of PERSIST_BATCH_SIZE = 25, pipelining database writes sequentially to cap memory pressure rather than buffering entire chunks in RAM.

Limiting Link Discovery Scope

To prevent unbounded memory growth from link extraction, the crawler implements dual ceilings:

• Stored links per page are capped at MAX_STORED_LINKS_PER_PAGE = 500
• Discovered URLs per batch are limited to MAX_DISCOVERED_PER_BATCH = 20,000

These boundaries in src/server/workflows/siteAuditWorkflowCrawl.ts ensure that pages with excessive navigation links or crawl traps cannot exhaust available memory or storage quotas.

Adaptive Concurrency and Memory Budgeting

OpenSEO optimizes throughput using an adaptive concurrency window defined in src/server/lib/audit/crawl-window.ts. The system initializes with INITIAL_CRAWL_WINDOW = 10 concurrent fetches and dynamically adjusts based on recent batch performance.

The adjustCrawlWindow function evaluates fetch success rates and response times, shrinking the window when encountering slow or error-prone pages and expanding it for fast responses. Crucially, the window respects an in-flight HTML budget of IN_FLIGHT_HTML_BUDGET_BYTES = 16 MiB, ensuring total buffered response bodies never exceed platform memory constraints.

Fault Tolerance via Progress Tracking

The architecture supports safe step replay through durable progress counters. Each crawl chunk reports attempted and pending metrics to AuditProgressKV, as implemented in src/server/lib/audit/progress-kv.ts.

If a chunk reports zero progress, the workflow triggers an early exit to prevent infinite loops on stuck domains. This stateless progress tracking allows the workflow to resume mid-audit without reprocessing completed chunks, essential for large sites requiring multiple worker invocations.

Selective Lighthouse Analysis

For the final Lighthouse phase, OpenSEO avoids comprehensive performance scans that would overwhelm large sites. As configured in src/server/workflows/siteAuditWorkflowPhases.ts, the LIGHTHOUSE_FETCH_STEP executes only on a representative sample of crawled pages rather than the full maxPages set. This sampling strategy delivers actionable Core Web Vitals data without proportional resource scaling.

Launching a Bounded Audit

To initiate an audit with custom limits:

import { AuditService } from "@/server/features/audit/services/AuditService";

export async function startAudit(
  projectId: string,
  startUrl: string,
  maxPages?: number,
) {
  // maxPages is automatically clamped to plan limits
  const auditId = await AuditService.start({
    projectId,
    startUrl,
    maxPages,
    lighthouseStrategy: "auto",
  });
  return auditId;
}

The AuditService.start method validates and clamps the maxPages parameter against tier limits before workflow invocation.

Summary

• Hard limits: The system enforces tier-specific maxPages caps in AuditService.ts before workflow execution begins.
• Scratchpad architecture: Discovery seeds URLs into a Cloudflare DO in 2,000-item batches, avoiding step-output size limits.
• Chunked crawling: The crawl phase processes 200-page chunks with 90-second soft deadlines and 25-item persistence batches.
• Memory safety: Link discovery is capped at 500 links per page and 20,000 discovered URLs per batch.
• Adaptive throughput: The rolling concurrency window adjusts between 10 and maximum safe concurrency while respecting a 16 MiB in-flight HTML budget.
• Fault tolerance: Progress tracking via KV store enables safe replays and early exit on stalled chunks.

Frequently Asked Questions

What is the maximum number of pages OpenSEO can audit in a single run?

The absolute maximum depends on the subscription tier, defined by PAID_MAX_AUDIT_PAGES and FREE_MAX_AUDIT_PAGES in src/server/lib/audit/types.ts. The AuditService clamps any user-provided maxPages value to these hard ceilings before the workflow initiates, ensuring the system never attempts to process more pages than the infrastructure supports.

How does OpenSEO prevent hitting Cloudflare Workers' step output limits?

Instead of returning discovered URLs in step outputs, the Discovery phase writes seed URLs directly to a durable scratchpad Durable Object. Batching seeds in groups of SEED_RPC_BATCH = 2,000 keeps individual RPC payloads well below the ~1 MiB step-output threshold, while the scratchpad frontier maintains state across workflow steps.

What happens if a website has pages with thousands of internal links?

The crawler implements safeguards in src/server/workflows/siteAuditWorkflowCrawl.ts by capping stored links at MAX_STORED_LINKS_PER_PAGE = 500 and discovered URLs per batch at MAX_DISCOVERED_PER_BATCH = 20,000. These limits prevent individual pages or crawl traps from consuming excessive memory or storage resources.

How does the system adapt to slow or unresponsive web servers?

The adjustCrawlWindow function in src/server/lib/audit/crawl-window.ts monitors recent batch performance and automatically shrinks the concurrency window when encountering slow responses or errors. This adaptive behavior, combined with the CHUNK_SOFT_DEADLINE_MS = 90,000 ms deadline per chunk, ensures workers complete within platform limits even when targeting unreliable infrastructure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}