# How the Open SEO Site Audit Workflow Uses Cloudflare Workflows for Crawling

> Discover how the Open SEO site audit workflow uses Cloudflare Workflows for robust crawling. Learn how it handles durable, retry-able steps and fetches pages concurrently. Read more now.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-08-07

---

**The Open SEO site audit workflow leverages Cloudflare Workflows to execute crawling as durable, retry-able steps, wrapping the `runCrawlPhase` function in `pgStep` to maintain state across isolated execution contexts while fetching up to 25 pages concurrently.**

The every-app/open-seo repository implements a comprehensive site auditing system that relies on Cloudflare Workflows to orchestrate complex, multi-phase crawls. By extending the `WorkflowEntrypoint` class and utilizing the `pgStep` helper for transactional durability, the site audit workflow Cloudflare Workflows crawling architecture ensures that each phase runs in its own execution context with automatic retry capabilities. This design allows the system to process thousands of URLs without losing progress when individual steps fail.

## Architecture Overview

The crawling process begins when the `AuditService` initiates a new workflow instance via `env.SITE_AUDIT_WORKFLOW.create()`. This invocation triggers the `SiteAuditWorkflow` class defined in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts), which serves as the entrypoint for all audit executions. The workflow orchestrates the entire audit lifecycle through three main stages: context validation, phase execution via `runAuditPhases`, and centralized error handling.

The `runAuditPhases` function imported from [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) coordinates sequential audit stages, including the critical crawl phase. Each phase executes as a discrete Cloudflare Workflow step, allowing the platform to persist state between steps and resume processing after interruptions.

## Workflow Entrypoint and Initialization

The `SiteAuditWorkflow` class extends Cloudflare's `WorkflowEntrypoint` and implements a `run` method that accepts the audit configuration. Inside [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts), the entrypoint first validates the request context using a durable `pgStep` to ensure the audit exists and belongs to the correct project.

```typescript
// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditWorkflowParams> {
  async run(event: WorkflowEvent<AuditWorkflowParams>, step: WorkflowStep) {
    // Validate context in a durable step
    const { audit, project } = await pgStep(step, "validate-context", undefined, async () => {
      return validateAuditContext(event.payload.auditId, event.payload.projectId);
    });
    
    // Execute audit phases
    await runAuditPhases(step, event, audit, project);
  }
}

```

The workflow instance is created in [`src/server/features/audit/services/AuditService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/services/AuditService.ts) using the binding exposed through the Cloudflare environment:

```typescript
// src/server/features/audit/services/AuditService.ts
await env.SITE_AUDIT_WORKFLOW.create({
  auditId,
  billingCustomer,
  projectId,
  startUrl,
  config,
});

```

## The Crawl Phase Implementation

The actual crawling logic resides in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) within the `runCrawlPhase` function. This function executes as a dedicated Cloudflare Workflow step wrapped by `pgStep`, ensuring that partial crawl progress persists even if the step encounters transient failures.

The crawl phase accepts parameters including the audit ID, workflow instance ID, origin domain, start URL, maximum page limits, robots.txt policy, and sitemap URLs. These parameters drive the frontier-based crawling algorithm that discovers and processes pages recursively.

```typescript
// src/server/workflows/siteAuditWorkflowCrawl.ts
export async function runCrawlPhase(
  step: WorkflowStep,
  params: CrawlPhaseParams
): Promise<CrawlPhaseResult> {
  return pgStep(step, "crawl", undefined, async () => {
    // Crawl implementation with durable state
    const { pages, isComplete } = await executeCrawl(params);
    return { pages, isComplete };
  });
}

```

## Concurrency and Frontier Management

The crawler maintains two distinct queues to manage URL discovery: one for link-discovered URLs and another for sitemap URLs. The implementation enforces a **concurrency limit of 25 parallel fetches** (`CRAWL_CONCURRENCY`) to respect target server resources while maximizing throughput.

To prevent memory exhaustion, the system caps new link discovery at **2,000 links per batch** (`MAX_FRONTIER_LINKS_PER_BATCH`). The `shouldQueueCrawlLink` function filters URLs through multiple validation layers including same-origin checks, crawlability heuristics, robots.txt policy enforcement via `robots.isAllowed()`, and deduplication against `visited` and `queued` sets.

```typescript
// src/server/workflows/siteAuditWorkflowCrawl.ts
const CRAWL_CONCURRENCY = 25;
const MAX_FRONTIER_LINKS_PER_BATCH = 2000;

while (linkQueue.length > 0 && summaries.length < maxPages) {
  const batch = linkQueue.splice(0, CRAWL_CONCURRENCY);
  const pageInfos = await Promise.all(
    batch.map(entry => crawlPage(entry.url, entry.depth, ...))
  );
  // Process results and enqueue new discoveries
}

```

## Page Fetching and Processing

Individual page fetching delegates to the `crawlPage` helper exported from [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts). This utility performs the actual HTTP fetch using Cloudflare's native `fetch` implementation available within Workers, executes Lighthouse audits when enabled, and extracts internal links for further discovery.

The helper returns a `StepPageSummary` object containing metadata about the crawled page, which accumulates into a `CrawledPageSummary[]` array passed to subsequent audit phases. Because Cloudflare Workflows isolate each step execution, the crawl step can safely store intermediate state in D1 or Durable Objects without risking data corruption during retries.

## Error Handling and Durability

The `pgStep` function defined in [`src/server/workflows/pgStep.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/pgStep.ts) wraps each workflow step in a transactional boundary, providing the durability guarantees essential for long-running crawls. If the `runCrawlPhase` step crashes due to network timeouts or transient errors, Cloudflare Workflows automatically retries the step without re-processing already-crawled pages.

Uncaught errors at the workflow level propagate to the main `run` method in `SiteAuditWorkflow`, which marks the audit as failed and persists the error state. This ensures that the system maintains data consistency even when facing infrastructure instability.

## Summary

- The **Open SEO** repository uses Cloudflare Workflows to execute site audits as a series of durable, isolated steps.
- The **`SiteAuditWorkflow`** class in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) extends `WorkflowEntrypoint` and orchestrates validation, crawling, and reporting phases.
- **Crawling** occurs within the `runCrawlPhase` function wrapped by `pgStep`, enabling automatic retries and state persistence.
- The crawler processes **25 pages concurrently** and limits new link discovery to **2,000 URLs per batch** to manage resource consumption.
- The **`crawlPage`** helper in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) handles individual page fetches using Cloudflare's Worker `fetch` API.

## Frequently Asked Questions

### What is the maximum concurrency for crawling in Open SEO?

The crawler enforces a hard concurrency limit of **25 simultaneous fetches** defined by the `CRAWL_CONCURRENCY` constant in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts). This limit balances throughput against server load and Cloudflare Worker resource constraints.

### How does the workflow handle failures during the crawl phase?

The `runCrawlPhase` function executes inside a `pgStep` wrapper that provides durable execution semantics. If a step fails due to network errors or timeouts, Cloudflare Workflows automatically retries the step from its last successful checkpoint without re-crawling pages already processed in previous attempts.

### What limits are placed on the URL frontier during crawling?

The implementation caps new link discovery at **2,000 links per batch** via `MAX_FRONTIER_LINKS_PER_BATCH` and maintains separate queues for link-discovered versus sitemap-discovered URLs. The `shouldQueueCrawlLink` function further filters URLs through same-origin validation, robots.txt policy checks, and deduplication logic.

### Which Cloudflare API does the site audit workflow extend?

The workflow extends **`WorkflowEntrypoint`** from the Cloudflare Workflows API, as implemented in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts). This base class provides the `run` method signature and step orchestration capabilities required for durable, long-running audit processes.