SiteAuditWorkflow Crawl Process Architecture: How Every-App Open-SEO Handles Large-Scale Site Audits

The SiteAuditWorkflow crawl process in open-seo is a fault-tolerant, chunk-based pipeline built on Cloudflare Workers Durable Objects that divides crawls into small, repeatable units to enable cheap retries and bounded memory usage.

The SiteAuditWorkflow crawl process powers comprehensive site audits in the every-app/open-seo repository. This architecture is specifically designed to operate within Cloudflare Workers' strict execution limits while handling sites with tens of thousands of pages. The system prioritizes resumability, idempotency, and resource awareness through a sophisticated chunking strategy and durable state management.

Core Components of the Crawl Architecture

The SiteAuditWorkflow crawl process is organized around several tightly integrated components, each with a distinct responsibility.

runAuditPhases: The High-Level Orchestrator

The runAuditPhases function in src/server/workflows/siteAuditWorkflowPhases.ts (lines 52-86) coordinates the entire audit lifecycle:

• Discovery phase: Seeds the frontier with start URLs and sitemap URLs
• Crawl phase: Executes the chunked crawl until completion or quota exhaustion
• Lighthouse phase: Runs performance audits on crawled pages
• Finalization phase: Multipage checks, link-edge validation, and cleanup

According to the every-app/open-seo source code, this orchestrator delegates crawl specifics to runCrawlPhase while managing cross-phase state through the audit scratchpad Durable Object.

runCrawlPhase: The Chunk Loop

Located in src/server/workflows/siteAuditWorkflowCrawl.ts (lines 86-122), runCrawlPhase implements the core iteration logic:

// Simplified conceptual flow from the actual implementation
async function runCrawlPhase(
  step,
  scratchpad,
  maxPages,
  origin,
  robots
) {
  let attempted = 0;
  let pending = 0;
  let chunkNo = 0;
  
  do {
    const result = await pgStep(
      `crawl-chunk-${chunkNo}`,
      () => runCrawlChunk(scratchpad, maxPages - attempted, chunkNo, origin, robots)
    );
    attempted += result.attemptedThisChunk;
    pending = result.pending;
    chunkNo++;
  } while (pending > 0 && attempted < maxPages);
  
  return { attempted, pending };
}

Each iteration calls pgStep to wrap chunk execution in a PostgreSQL-backed checkpoint, guaranteeing idempotent retries on failure.

runCrawlChunk: The Execution Unit

The runCrawlChunk function (lines 124-443 in src/server/workflows/siteAuditWorkflowCrawl.ts) handles the actual work of a single chunk:

1. Claims URLs from the scratchpad via scratchpad.claimChunk
2. Fetches pages using a dynamic concurrency window that adapts to performance
3. Respects back-pressure by monitoring pending persistence batches
4. Releases un-attempted URLs if the soft deadline fires

Chunk Sizing and Resource Limits

The SiteAuditWorkflow crawl process enforces strict bounds to prevent resource exhaustion. These constants are defined at lines 32-58 of siteAuditWorkflowCrawl.ts:

Constant	Value	Purpose
CHUNK_TARGET_PAGES	200	Maximum URLs processed per chunk
CHUNK_SOFT_DEADLINE_MS	90,000	Stops new fetches after ~90 seconds
PERSIST_BATCH_SIZE	25	Pages flushed to database in small batches
MAX_QUEUED_PERSIST_BATCHES	2	Caps pending write operations
MAX_STORED_LINKS_PER_PAGE	500	Limits stored link rows per page
MAX_DISCOVERED_PER_BATCH	20,000	Caps newly discovered URLs per batch

These limits ensure the SiteAuditWorkflow crawl process remains safe within Cloudflare Workers' 128MB memory limit and 30-second CPU time restrictions (extended to minutes for Durable Objects with proper yielding).

State Management with the Audit Scratchpad

All crawl state lives in the AuditScratchpad Durable Object, not in step payloads. This design is critical for fault tolerance.

Key Scratchpad Operations

• claimChunk(chunkNo, limit): Atomically reserves URLs for processing
• recordBatch(links, discoveredUrls): Stores newly found internal links and frontier additions
• Lease management: Tracks which URLs are currently being processed

The scratchpad implementation in src/server/features/audit/AuditScratchpad.ts uses Durable Object storage for:

• The URL frontier (pending crawl targets)
• Claimed URL leases (in-progress work)
• Discovered links (for graph analysis)
• Progress counters (attempted, pending, completed)

Persistence and Idempotency with pgStep

Every durable step wraps execution in pgStep from src/server/workflows/pgStep.ts. This utility:

1. Checks PostgreSQL for a completed checkpoint before executing
2. Runs the step function if no checkpoint exists
3. Stores the result atomically on success

For idempotent writes, the system generates deterministic row IDs via deterministicAuditRowId (used in persistCrawledPages, line 58). This ensures that retried steps produce identical database rows without duplicates.

Link Discovery and Queueing Logic

After fetching a page, the SiteAuditWorkflow crawl process examines internal links through two key functions in siteAuditWorkflowCrawl.ts:

• shouldQueueCrawlLink (lines 60-68): Filters links by same-origin, crawl-allowed status, and exclusion rules
• Discovery loop in persistCrawledPages (lines 70-89): Adds valid links to the scratchpad frontier

// Conceptual flow from the actual implementation
function shouldQueueCrawlLink(link, origin, robots) {
  if (!isSameOrigin(link, origin)) return false;
  if (isDisallowedByRobots(link, robots)) return false;
  if (isAlreadyQueued(link)) return false;
  return true;
}

Progress Reporting for Real-Time UI

The AuditProgressKV.pushCrawledUrls function (lines 19-27 of the persistence module) updates a Cloudflare KV store with crawl progress after each persisted sub-batch. This enables live UI updates showing:

• Pages crawled
• Issues found
• Estimated completion

Fault-Tolerance Mechanisms

The SiteAuditWorkflow crawl process implements multiple resilience strategies:

• Step-level retries: pgStep automatically retries failed chunks
• URL lease expiration: Unfinished URLs are released and reclaimed on retry
• Soft deadline enforcement: Prevents chunks from exceeding execution limits
• Back-pressure throttling: Halts new fetches when persistence queues fill
• Deterministic IDs: Eliminates duplicate data on re-execution

Complete Crawl Flow Example

// Initiate a full site audit
import { runAuditPhases } from './src/server/workflows/siteAuditWorkflowPhases';

await runAuditPhases(step, {
  auditId: 'audit_123',
  workflowInstanceId: 'wf_456',
  billingCustomer: 'cust_789',
  projectId: 'proj_abc',
  startUrl: 'https://example.com',
  config: {
    maxPages: 5000,
    lighthouseSampleRate: 0.1,
    // ... additional configuration
  },
});

The SiteAuditWorkflow crawl process then executes autonomously, with the scratchpad maintaining all intermediate state.

Summary

• The SiteAuditWorkflow crawl process uses 200-page chunks to divide large crawls into manageable, retryable units
• All state persists in a Cloudflare Workers Durable Object (AuditScratchpad), not in step memory
• pgStep wraps every chunk in PostgreSQL checkpoints for guaranteed idempotency
• Dynamic concurrency windows and back-pressure limits prevent resource exhaustion
• Deterministic row IDs ensure duplicate-free database writes across retries
• 90-second soft deadlines and URL lease management enable graceful degradation under pressure

Frequently Asked Questions

How does the SiteAuditWorkflow crawl process handle worker crashes mid-crawl?

The scratchpad Durable Object maintains URL leases separately from execution state. When a worker crashes, leases eventually expire and their URLs return to the frontier. On retry, pgStep detects the incomplete checkpoint and re-executes the chunk, with idempotent writes preventing duplicates.

Why does every-app/open-seo use 200-page chunks instead of larger batches?

The CHUNK_TARGET_PAGES = 200 constant balances throughput against Cloudflare Workers constraints. Smaller chunks mean faster persistence, lower memory usage, and cheaper retries. Larger batches risk hitting the 128MB memory limit or 30-second CPU time on complex pages.

Where is crawl progress stored during execution?

Progress lives exclusively in the AuditScratchpad Durable Object and PostgreSQL checkpoints via pgStep. Real-time UI updates use AuditProgressKV.pushCrawledUrls to write summaries to Cloudflare KV, but this is a secondary reporting channel—the source of truth remains the scratchpad.

How does the crawl process prevent duplicate page records?

The deterministicAuditRowId function generates reproducible identifiers from audit ID, URL, and crawl depth. Whether a page is fetched once or retried multiple times, the same ID is computed, causing PostgreSQL INSERT operations to be naturally idempotent (or handled via upsert logic in AuditRepository.insertCrawledBatch).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}