# SiteAuditWorkflow Crawl Process Architecture: How Every-App Open-SEO Handles Large-Scale Site Audits

> Explore the SiteAuditWorkflow crawl process architecture in every-app/open-seo. Learn how Durable Objects handle large-scale site audits with fault-tolerant, chunk-based pipelines for efficient retries and memory management.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: architecture
- Published: 2026-08-10

---

**The SiteAuditWorkflow crawl process in open-seo is a fault-tolerant, chunk-based pipeline built on Cloudflare Workers Durable Objects that divides crawls into small, repeatable units to enable cheap retries and bounded memory usage.**

The **SiteAuditWorkflow crawl process** powers comprehensive site audits in the [every-app/open-seo](https://github.com/every-app/open-seo) repository. This architecture is specifically designed to operate within Cloudflare Workers' strict execution limits while handling sites with tens of thousands of pages. The system prioritizes **resumability**, **idempotency**, and **resource awareness** through a sophisticated chunking strategy and durable state management.

## Core Components of the Crawl Architecture

The SiteAuditWorkflow crawl process is organized around several tightly integrated components, each with a distinct responsibility.

### runAuditPhases: The High-Level Orchestrator

The `runAuditPhases` function in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) (lines 52-86) coordinates the entire audit lifecycle:

- **Discovery phase**: Seeds the frontier with start URLs and sitemap URLs
- **Crawl phase**: Executes the chunked crawl until completion or quota exhaustion
- **Lighthouse phase**: Runs performance audits on crawled pages
- **Finalization phase**: Multipage checks, link-edge validation, and cleanup

According to the every-app/open-seo source code, this orchestrator delegates crawl specifics to `runCrawlPhase` while managing cross-phase state through the **audit scratchpad** Durable Object.

### runCrawlPhase: The Chunk Loop

Located in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) (lines 86-122), `runCrawlPhase` implements the core iteration logic:

```typescript
// Simplified conceptual flow from the actual implementation
async function runCrawlPhase(
  step,
  scratchpad,
  maxPages,
  origin,
  robots
) {
  let attempted = 0;
  let pending = 0;
  let chunkNo = 0;
  
  do {
    const result = await pgStep(
      `crawl-chunk-${chunkNo}`,
      () => runCrawlChunk(scratchpad, maxPages - attempted, chunkNo, origin, robots)
    );
    attempted += result.attemptedThisChunk;
    pending = result.pending;
    chunkNo++;
  } while (pending > 0 && attempted < maxPages);
  
  return { attempted, pending };
}

```

Each iteration calls `pgStep` to wrap chunk execution in a PostgreSQL-backed checkpoint, guaranteeing **idempotent retries** on failure.

### runCrawlChunk: The Execution Unit

The `runCrawlChunk` function (lines 124-443 in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts)) handles the actual work of a single chunk:

1. **Claims URLs** from the scratchpad via `scratchpad.claimChunk`
2. **Fetches pages** using a dynamic concurrency window that adapts to performance
3. **Respects back-pressure** by monitoring pending persistence batches
4. **Releases un-attempted URLs** if the soft deadline fires

## Chunk Sizing and Resource Limits

The SiteAuditWorkflow crawl process enforces strict bounds to prevent resource exhaustion. These constants are defined at lines 32-58 of [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts):

| Constant | Value | Purpose |
|----------|-------|---------|
| `CHUNK_TARGET_PAGES` | 200 | Maximum URLs processed per chunk |
| `CHUNK_SOFT_DEADLINE_MS` | 90,000 | Stops new fetches after ~90 seconds |
| `PERSIST_BATCH_SIZE` | 25 | Pages flushed to database in small batches |
| `MAX_QUEUED_PERSIST_BATCHES` | 2 | Caps pending write operations |
| `MAX_STORED_LINKS_PER_PAGE` | 500 | Limits stored link rows per page |
| `MAX_DISCOVERED_PER_BATCH` | 20,000 | Caps newly discovered URLs per batch |

These limits ensure the SiteAuditWorkflow crawl process remains safe within **Cloudflare Workers' 128MB memory limit and 30-second CPU time restrictions** (extended to minutes for Durable Objects with proper yielding).

## State Management with the Audit Scratchpad

All crawl state lives in the **AuditScratchpad** Durable Object, not in step payloads. This design is critical for fault tolerance.

### Key Scratchpad Operations

- `claimChunk(chunkNo, limit)`: Atomically reserves URLs for processing
- `recordBatch(links, discoveredUrls)`: Stores newly found internal links and frontier additions
- Lease management: Tracks which URLs are currently being processed

The scratchpad implementation in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts) uses Durable Object storage for:

- The **URL frontier** (pending crawl targets)
- **Claimed URL leases** (in-progress work)
- **Discovered links** (for graph analysis)
- **Progress counters** (attempted, pending, completed)

## Persistence and Idempotency with pgStep

Every durable step wraps execution in `pgStep` from [`src/server/workflows/pgStep.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/pgStep.ts). This utility:

1. Checks PostgreSQL for a completed checkpoint before executing
2. Runs the step function if no checkpoint exists
3. Stores the result atomically on success

For idempotent writes, the system generates **deterministic row IDs** via `deterministicAuditRowId` (used in `persistCrawledPages`, line 58). This ensures that retried steps produce identical database rows without duplicates.

## Link Discovery and Queueing Logic

After fetching a page, the SiteAuditWorkflow crawl process examines internal links through two key functions in [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts):

- `shouldQueueCrawlLink` (lines 60-68): Filters links by same-origin, crawl-allowed status, and exclusion rules
- Discovery loop in `persistCrawledPages` (lines 70-89): Adds valid links to the scratchpad frontier

```typescript
// Conceptual flow from the actual implementation
function shouldQueueCrawlLink(link, origin, robots) {
  if (!isSameOrigin(link, origin)) return false;
  if (isDisallowedByRobots(link, robots)) return false;
  if (isAlreadyQueued(link)) return false;
  return true;
}

```

## Progress Reporting for Real-Time UI

The `AuditProgressKV.pushCrawledUrls` function (lines 19-27 of the persistence module) updates a **Cloudflare KV store** with crawl progress after each persisted sub-batch. This enables live UI updates showing:

- Pages crawled
- Issues found
- Estimated completion

## Fault-Tolerance Mechanisms

The SiteAuditWorkflow crawl process implements multiple resilience strategies:

- **Step-level retries**: `pgStep` automatically retries failed chunks
- **URL lease expiration**: Unfinished URLs are released and reclaimed on retry
- **Soft deadline enforcement**: Prevents chunks from exceeding execution limits
- **Back-pressure throttling**: Halts new fetches when persistence queues fill
- **Deterministic IDs**: Eliminates duplicate data on re-execution

## Complete Crawl Flow Example

```typescript
// Initiate a full site audit
import { runAuditPhases } from './src/server/workflows/siteAuditWorkflowPhases';

await runAuditPhases(step, {
  auditId: 'audit_123',
  workflowInstanceId: 'wf_456',
  billingCustomer: 'cust_789',
  projectId: 'proj_abc',
  startUrl: 'https://example.com',
  config: {
    maxPages: 5000,
    lighthouseSampleRate: 0.1,
    // ... additional configuration
  },
});

```

The SiteAuditWorkflow crawl process then executes autonomously, with the scratchpad maintaining all intermediate state.

## Summary

- The **SiteAuditWorkflow crawl process** uses **200-page chunks** to divide large crawls into manageable, retryable units
- All state persists in a **Cloudflare Workers Durable Object** (AuditScratchpad), not in step memory
- **`pgStep`** wraps every chunk in PostgreSQL checkpoints for guaranteed idempotency
- **Dynamic concurrency windows** and **back-pressure limits** prevent resource exhaustion
- **Deterministic row IDs** ensure duplicate-free database writes across retries
- **90-second soft deadlines** and **URL lease management** enable graceful degradation under pressure

## Frequently Asked Questions

### How does the SiteAuditWorkflow crawl process handle worker crashes mid-crawl?

The scratchpad Durable Object maintains URL leases separately from execution state. When a worker crashes, leases eventually expire and their URLs return to the frontier. On retry, `pgStep` detects the incomplete checkpoint and re-executes the chunk, with idempotent writes preventing duplicates.

### Why does every-app/open-seo use 200-page chunks instead of larger batches?

The `CHUNK_TARGET_PAGES = 200` constant balances throughput against Cloudflare Workers constraints. Smaller chunks mean faster persistence, lower memory usage, and cheaper retries. Larger batches risk hitting the 128MB memory limit or 30-second CPU time on complex pages.

### Where is crawl progress stored during execution?

Progress lives exclusively in the **AuditScratchpad** Durable Object and **PostgreSQL checkpoints** via `pgStep`. Real-time UI updates use `AuditProgressKV.pushCrawledUrls` to write summaries to Cloudflare KV, but this is a secondary reporting channel—the source of truth remains the scratchpad.

### How does the crawl process prevent duplicate page records?

The `deterministicAuditRowId` function generates reproducible identifiers from audit ID, URL, and crawl depth. Whether a page is fetched once or retried multiple times, the same ID is computed, causing PostgreSQL `INSERT` operations to be naturally idempotent (or handled via upsert logic in `AuditRepository.insertCrawledBatch`).