How Open SEO Implements Durable Cloudflare Workflows for Site Crawling

The SiteAuditWorkflow in every-app/open-seo implements durable Cloudflare Workflows for site crawling by extending WorkflowEntrypoint, wrapping every heavy operation in checkpointed pgStep calls, and storing mutable crawl state inside an AuditScratchpad Durable Object so that retries resume from the last successful step instead of restarting the entire audit.

The every-app/open-seo repository is a production-grade site auditing engine built entirely on Cloudflare's edge runtime. Its SiteAuditWorkflow demonstrates how to implement durable Cloudflare Workflows for site crawling by decomposing a long-running audit into discrete, replay-safe phases that persist progress automatically through worker crashes and code redeployments.

Durable Entrypoint with WorkflowEntrypoint

In src/server/workflows/SiteAuditWorkflow.ts (lines 30‑38), the SiteAuditWorkflow class extends WorkflowEntrypoint<Env, AuditParams>. This base class scopes the audit to a single Durable Object instance, which means every retry or replay restarts the same object rather than spawning a fresh worker. The run method opens a per-request Postgres client through withPgClient and then delegates to this.runScoped(event, step), ensuring database access is replay-safe.

// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    // Scope a per‑request DB client (no‑op on D1) and delegate to the
    // durable‑step implementation.
    return withPgClient(() => this.runScoped(event, step));
  }
}

Checkpointed Steps via pgStep

All heavy work is wrapped in pgStep from src/server/workflows/pgStep.ts. This utility records a durable checkpoint for the step name before executing the callback. If the workflow is interrupted, the Cloudflare runtime rewinds to the last successful checkpoint and re-executes only the failed step. The configuration constants DB_STEP and CRAWL_CHUNK_STEP are defined in src/server/workflows/auditStepConfigs.ts, giving each step a stable identity that the runtime can reference during replay.

Discovery Phase and AuditScratchpad Caching

The first checkpointed phase is discovery. In src/server/workflows/siteAuditWorkflowPhases.ts, the runDiscoveryPhase function (lines 21‑34) executes under the step name discover-urls-v2. It fetches the site's sitemap and robots.txt once, then stores the resulting seed URLs directly in the AuditScratchpad Durable Object. Because pgStep checkpoints this result, subsequent replays reuse the cached discovery data instead of reissuing HTTP requests.

Chunked Crawling for O(1) Memory

The actual crawl is split into chunks to respect Durable Object limits. In src/server/workflows/siteAuditWorkflowCrawl.ts, the runCrawlChunk function (lines 24‑44) leases up to CHUNK_TARGET_PAGES URLs from the scratchpad, fetches them concurrently, and persists results in small batches sized by PERSIST_BATCH_SIZE. Each chunk is wrapped in its own pgStep call with a stable name like crawl-chunk-N.

This design keeps individual step state under 1 MiB, which means the workflow heap stays O(1) regardless of how large the target site is. If crawl-chunk-3 fails due to a transient error, only that chunk retries.

// src/server/workflows/siteAuditWorkflowCrawl.ts
const result = await pgStep(
  step,
  `crawl-chunk-${chunkNo}`,
  CRAWL_CHUNK_STEP,
  () => runCrawlChunk({ ...params, chunkNo, attemptedBefore: attemptedTotal })
);

Idempotent Persistence and Progress Tracking

Inside persistCrawledPages (src/server/workflows/siteAuditWorkflowCrawl.ts, lines 15‑19), every page receives a deterministic ID through deterministicAuditRowId from src/server/lib/audit/ids.ts. This makes AuditRepository.insertCrawledBatch idempotent across replays, so duplicate rows are never created when Cloudflare Workflows re-executes a step.

Each chunk also updates global counters via AuditProgressKV and AuditRepository.updateAuditProgress. These counters are backed by durable storage, which guarantees that a replay sees the exact same progress numbers even if the database was temporarily unavailable during the previous attempt.

// src/server/workflows/siteAuditWorkflowCrawl.ts (persistCrawledPages)
for (const page of pages) {
  page.id = await deterministicAuditRowId(auditId, page.url);
}
await AuditRepository.insertCrawledBatch(auditId, pages, issues);

Lighthouse and Finalization Checkpoints

Later phases follow the same checkpoint pattern. In src/server/workflows/siteAuditWorkflowPhases.ts, the runLighthousePhase function (lines 7‑15) wraps Lighthouse fetches in pgStep under names such as lighthouse-batch-X. Because these expensive external calls use the DB_STEP configuration, they are performed exactly once unless the specific batch was interrupted. Finalization and multipage checks are wrapped identically.

// src/server/workflows/siteAuditWorkflowPhases.ts
const boundary = await pgStep(
  step,
  `lighthouse-batch-${batchIndex + 1}`,
  DB_STEP,
  async (): Promise<LighthouseBatchBoundary> => ({ schema: "retry-safe-v2" })
);

Summary

• WorkflowEntrypoint in SiteAuditWorkflow.ts anchors every audit to a single Durable Object instance.
• pgStep checkpoints each phase so that retries resume from the last successful step.
• Chunked crawling caps memory usage and isolates failures to individual crawl-chunk-N steps.
• AuditScratchpad stores the URL frontier and lease state in durable Cloudflare DO storage.
• deterministicAuditRowId guarantees idempotent database writes across replays.

Frequently Asked Questions

What makes the SiteAuditWorkflow durable?

The workflow achieves durability by combining three mechanisms: it extends WorkflowEntrypoint to run inside a single Durable Object, wraps every operation in pgStep so the runtime can replay from the last checkpoint, and stores all crawl frontier data in the AuditScratchpad Durable Object. Together, these ensure that crashes or redeployments never lose progress.

How does pgStep prevent duplicate work on retries?

pgStep writes a durable checkpoint under a stable step name before executing the callback. If the workflow replays, the runtime checks whether that step already completed; if so, it returns the cached result instead of re-executing the callback. This means discovery, crawl chunks, and Lighthouse fetches run exactly once unless the step itself was interrupted.

Why is the crawl split into small chunks?

Crawling in chunks keeps each step's state under the Durable Object limit—typically less than 1 MiB—and ensures the workflow heap remains O(1) regardless of site size. It also limits the blast radius of failures: if one chunk fails, Cloudflare Workflows retries only that crawl-chunk-N rather than the entire site.

How does the AuditScratchpad survive worker restarts?

AuditScratchpad is a Cloudflare Durable Object defined in src/server/features/audit/AuditScratchpad.ts. Unlike ephemeral worker memory, Durable Objects are backed by Cloudflare's storage layer. The scratchpad tracks claimed URLs through scratchpad.claimChunk and releases any unlaunched URLs back to the frontier on failure, guaranteeing no work is lost across restarts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}