# How Open SEO Implements Durable Cloudflare Workflows for Site Crawling

> Discover how Open SEO's SiteAuditWorkflow uses durable Cloudflare Workflows and Durable Objects to enable resilient site crawling with automatic retry from the last successful step.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-08-19

---

**The SiteAuditWorkflow in `every-app/open-seo` implements durable Cloudflare Workflows for site crawling by extending `WorkflowEntrypoint`, wrapping every heavy operation in checkpointed `pgStep` calls, and storing mutable crawl state inside an `AuditScratchpad` Durable Object so that retries resume from the last successful step instead of restarting the entire audit.**

The `every-app/open-seo` repository is a production-grade site auditing engine built entirely on Cloudflare's edge runtime. Its `SiteAuditWorkflow` demonstrates how to implement durable Cloudflare Workflows for site crawling by decomposing a long-running audit into discrete, replay-safe phases that persist progress automatically through worker crashes and code redeployments.

## Durable Entrypoint with WorkflowEntrypoint

In [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) (lines 30‑38), the **SiteAuditWorkflow** class extends **WorkflowEntrypoint<Env, AuditParams>**. This base class scopes the audit to a single **Durable Object** instance, which means every retry or replay restarts the same object rather than spawning a fresh worker. The `run` method opens a per-request Postgres client through **withPgClient** and then delegates to `this.runScoped(event, step)`, ensuring database access is replay-safe.

```ts
// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    // Scope a per‑request DB client (no‑op on D1) and delegate to the
    // durable‑step implementation.
    return withPgClient(() => this.runScoped(event, step));
  }
}

```

## Checkpointed Steps via pgStep

All heavy work is wrapped in **pgStep** from [`src/server/workflows/pgStep.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/pgStep.ts). This utility records a durable checkpoint for the step name before executing the callback. If the workflow is interrupted, the Cloudflare runtime rewinds to the last successful checkpoint and re-executes only the failed step. The configuration constants **DB_STEP** and **CRAWL_CHUNK_STEP** are defined in [`src/server/workflows/auditStepConfigs.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/auditStepConfigs.ts), giving each step a stable identity that the runtime can reference during replay.

## Discovery Phase and AuditScratchpad Caching

The first checkpointed phase is discovery. In [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), the `runDiscoveryPhase` function (lines 21‑34) executes under the step name `discover-urls-v2`. It fetches the site's sitemap and robots.txt once, then stores the resulting seed URLs directly in the **AuditScratchpad** Durable Object. Because `pgStep` checkpoints this result, subsequent replays reuse the cached discovery data instead of reissuing HTTP requests.

## Chunked Crawling for O(1) Memory

The actual crawl is split into chunks to respect Durable Object limits. In [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), the `runCrawlChunk` function (lines 24‑44) leases up to `CHUNK_TARGET_PAGES` URLs from the scratchpad, fetches them concurrently, and persists results in small batches sized by `PERSIST_BATCH_SIZE`. Each chunk is wrapped in its own `pgStep` call with a stable name like `crawl-chunk-N`.

This design keeps individual step state under 1 MiB, which means the workflow heap stays **O(1)** regardless of how large the target site is. If `crawl-chunk-3` fails due to a transient error, only that chunk retries.

```ts
// src/server/workflows/siteAuditWorkflowCrawl.ts
const result = await pgStep(
  step,
  `crawl-chunk-${chunkNo}`,
  CRAWL_CHUNK_STEP,
  () => runCrawlChunk({ ...params, chunkNo, attemptedBefore: attemptedTotal })
);

```

## Idempotent Persistence and Progress Tracking

Inside `persistCrawledPages` ([`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), lines 15‑19), every page receives a deterministic ID through **deterministicAuditRowId** from [`src/server/lib/audit/ids.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/ids.ts). This makes `AuditRepository.insertCrawledBatch` idempotent across replays, so duplicate rows are never created when Cloudflare Workflows re-executes a step.

Each chunk also updates global counters via `AuditProgressKV` and `AuditRepository.updateAuditProgress`. These counters are backed by durable storage, which guarantees that a replay sees the exact same progress numbers even if the database was temporarily unavailable during the previous attempt.

```ts
// src/server/workflows/siteAuditWorkflowCrawl.ts (persistCrawledPages)
for (const page of pages) {
  page.id = await deterministicAuditRowId(auditId, page.url);
}
await AuditRepository.insertCrawledBatch(auditId, pages, issues);

```

## Lighthouse and Finalization Checkpoints

Later phases follow the same checkpoint pattern. In [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), the `runLighthousePhase` function (lines 7‑15) wraps Lighthouse fetches in `pgStep` under names such as `lighthouse-batch-X`. Because these expensive external calls use the `DB_STEP` configuration, they are performed exactly once unless the specific batch was interrupted. Finalization and multipage checks are wrapped identically.

```ts
// src/server/workflows/siteAuditWorkflowPhases.ts
const boundary = await pgStep(
  step,
  `lighthouse-batch-${batchIndex + 1}`,
  DB_STEP,
  async (): Promise<LighthouseBatchBoundary> => ({ schema: "retry-safe-v2" })
);

```

## Summary

- **WorkflowEntrypoint** in [`SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/SiteAuditWorkflow.ts) anchors every audit to a single Durable Object instance.
- **pgStep** checkpoints each phase so that retries resume from the last successful step.
- **Chunked crawling** caps memory usage and isolates failures to individual `crawl-chunk-N` steps.
- **AuditScratchpad** stores the URL frontier and lease state in durable Cloudflare DO storage.
- **deterministicAuditRowId** guarantees idempotent database writes across replays.

## Frequently Asked Questions

### What makes the SiteAuditWorkflow durable?

The workflow achieves durability by combining three mechanisms: it extends `WorkflowEntrypoint` to run inside a single Durable Object, wraps every operation in `pgStep` so the runtime can replay from the last checkpoint, and stores all crawl frontier data in the `AuditScratchpad` Durable Object. Together, these ensure that crashes or redeployments never lose progress.

### How does pgStep prevent duplicate work on retries?

`pgStep` writes a durable checkpoint under a stable step name before executing the callback. If the workflow replays, the runtime checks whether that step already completed; if so, it returns the cached result instead of re-executing the callback. This means discovery, crawl chunks, and Lighthouse fetches run exactly once unless the step itself was interrupted.

### Why is the crawl split into small chunks?

Crawling in chunks keeps each step's state under the Durable Object limit—typically less than 1 MiB—and ensures the workflow heap remains **O(1)** regardless of site size. It also limits the blast radius of failures: if one chunk fails, Cloudflare Workflows retries only that `crawl-chunk-N` rather than the entire site.

### How does the AuditScratchpad survive worker restarts?

`AuditScratchpad` is a Cloudflare Durable Object defined in [`src/server/features/audit/AuditScratchpad.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/AuditScratchpad.ts). Unlike ephemeral worker memory, Durable Objects are backed by Cloudflare's storage layer. The scratchpad tracks claimed URLs through `scratchpad.claimChunk` and releases any unlaunched URLs back to the frontier on failure, guaranteeing no work is lost across restarts.