How the Open SEO Site Audit Workflow Uses Cloudflare Workflows for Crawling

The Open SEO site audit workflow leverages Cloudflare Workflows to execute crawling as durable, retry-able steps, wrapping the runCrawlPhase function in pgStep to maintain state across isolated execution contexts while fetching up to 25 pages concurrently.

The every-app/open-seo repository implements a comprehensive site auditing system that relies on Cloudflare Workflows to orchestrate complex, multi-phase crawls. By extending the WorkflowEntrypoint class and utilizing the pgStep helper for transactional durability, the site audit workflow Cloudflare Workflows crawling architecture ensures that each phase runs in its own execution context with automatic retry capabilities. This design allows the system to process thousands of URLs without losing progress when individual steps fail.

Architecture Overview

The crawling process begins when the AuditService initiates a new workflow instance via env.SITE_AUDIT_WORKFLOW.create(). This invocation triggers the SiteAuditWorkflow class defined in src/server/workflows/SiteAuditWorkflow.ts, which serves as the entrypoint for all audit executions. The workflow orchestrates the entire audit lifecycle through three main stages: context validation, phase execution via runAuditPhases, and centralized error handling.

The runAuditPhases function imported from src/server/workflows/siteAuditWorkflowPhases.ts coordinates sequential audit stages, including the critical crawl phase. Each phase executes as a discrete Cloudflare Workflow step, allowing the platform to persist state between steps and resume processing after interruptions.

Workflow Entrypoint and Initialization

The SiteAuditWorkflow class extends Cloudflare's WorkflowEntrypoint and implements a run method that accepts the audit configuration. Inside src/server/workflows/SiteAuditWorkflow.ts, the entrypoint first validates the request context using a durable pgStep to ensure the audit exists and belongs to the correct project.

// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditWorkflowParams> {
  async run(event: WorkflowEvent<AuditWorkflowParams>, step: WorkflowStep) {
    // Validate context in a durable step
    const { audit, project } = await pgStep(step, "validate-context", undefined, async () => {
      return validateAuditContext(event.payload.auditId, event.payload.projectId);
    });
    
    // Execute audit phases
    await runAuditPhases(step, event, audit, project);
  }
}

The workflow instance is created in src/server/features/audit/services/AuditService.ts using the binding exposed through the Cloudflare environment:

// src/server/features/audit/services/AuditService.ts
await env.SITE_AUDIT_WORKFLOW.create({
  auditId,
  billingCustomer,
  projectId,
  startUrl,
  config,
});

The Crawl Phase Implementation

The actual crawling logic resides in src/server/workflows/siteAuditWorkflowCrawl.ts within the runCrawlPhase function. This function executes as a dedicated Cloudflare Workflow step wrapped by pgStep, ensuring that partial crawl progress persists even if the step encounters transient failures.

The crawl phase accepts parameters including the audit ID, workflow instance ID, origin domain, start URL, maximum page limits, robots.txt policy, and sitemap URLs. These parameters drive the frontier-based crawling algorithm that discovers and processes pages recursively.

// src/server/workflows/siteAuditWorkflowCrawl.ts
export async function runCrawlPhase(
  step: WorkflowStep,
  params: CrawlPhaseParams
): Promise<CrawlPhaseResult> {
  return pgStep(step, "crawl", undefined, async () => {
    // Crawl implementation with durable state
    const { pages, isComplete } = await executeCrawl(params);
    return { pages, isComplete };
  });
}

Concurrency and Frontier Management

The crawler maintains two distinct queues to manage URL discovery: one for link-discovered URLs and another for sitemap URLs. The implementation enforces a concurrency limit of 25 parallel fetches (CRAWL_CONCURRENCY) to respect target server resources while maximizing throughput.

To prevent memory exhaustion, the system caps new link discovery at 2,000 links per batch (MAX_FRONTIER_LINKS_PER_BATCH). The shouldQueueCrawlLink function filters URLs through multiple validation layers including same-origin checks, crawlability heuristics, robots.txt policy enforcement via robots.isAllowed(), and deduplication against visited and queued sets.

// src/server/workflows/siteAuditWorkflowCrawl.ts
const CRAWL_CONCURRENCY = 25;
const MAX_FRONTIER_LINKS_PER_BATCH = 2000;

while (linkQueue.length > 0 && summaries.length < maxPages) {
  const batch = linkQueue.splice(0, CRAWL_CONCURRENCY);
  const pageInfos = await Promise.all(
    batch.map(entry => crawlPage(entry.url, entry.depth, ...))
  );
  // Process results and enqueue new discoveries
}

Page Fetching and Processing

Individual page fetching delegates to the crawlPage helper exported from src/server/workflows/site-audit-workflow-helpers.ts. This utility performs the actual HTTP fetch using Cloudflare's native fetch implementation available within Workers, executes Lighthouse audits when enabled, and extracts internal links for further discovery.

The helper returns a StepPageSummary object containing metadata about the crawled page, which accumulates into a CrawledPageSummary[] array passed to subsequent audit phases. Because Cloudflare Workflows isolate each step execution, the crawl step can safely store intermediate state in D1 or Durable Objects without risking data corruption during retries.

Error Handling and Durability

The pgStep function defined in src/server/workflows/pgStep.ts wraps each workflow step in a transactional boundary, providing the durability guarantees essential for long-running crawls. If the runCrawlPhase step crashes due to network timeouts or transient errors, Cloudflare Workflows automatically retries the step without re-processing already-crawled pages.

Uncaught errors at the workflow level propagate to the main run method in SiteAuditWorkflow, which marks the audit as failed and persists the error state. This ensures that the system maintains data consistency even when facing infrastructure instability.

Summary

• The Open SEO repository uses Cloudflare Workflows to execute site audits as a series of durable, isolated steps.
• The SiteAuditWorkflow class in src/server/workflows/SiteAuditWorkflow.ts extends WorkflowEntrypoint and orchestrates validation, crawling, and reporting phases.
• Crawling occurs within the runCrawlPhase function wrapped by pgStep, enabling automatic retries and state persistence.
• The crawler processes 25 pages concurrently and limits new link discovery to 2,000 URLs per batch to manage resource consumption.
• The crawlPage helper in src/server/workflows/site-audit-workflow-helpers.ts handles individual page fetches using Cloudflare's Worker fetch API.

Frequently Asked Questions

What is the maximum concurrency for crawling in Open SEO?

The crawler enforces a hard concurrency limit of 25 simultaneous fetches defined by the CRAWL_CONCURRENCY constant in src/server/workflows/siteAuditWorkflowCrawl.ts. This limit balances throughput against server load and Cloudflare Worker resource constraints.

How does the workflow handle failures during the crawl phase?

The runCrawlPhase function executes inside a pgStep wrapper that provides durable execution semantics. If a step fails due to network errors or timeouts, Cloudflare Workflows automatically retries the step from its last successful checkpoint without re-crawling pages already processed in previous attempts.

What limits are placed on the URL frontier during crawling?

The implementation caps new link discovery at 2,000 links per batch via MAX_FRONTIER_LINKS_PER_BATCH and maintains separate queues for link-discovered versus sitemap-discovered URLs. The shouldQueueCrawlLink function further filters URLs through same-origin validation, robots.txt policy checks, and deduplication logic.

Which Cloudflare API does the site audit workflow extend?

The workflow extends WorkflowEntrypoint from the Cloudflare Workflows API, as implemented in src/server/workflows/SiteAuditWorkflow.ts. This base class provides the run method signature and step orchestration capabilities required for durable, long-running audit processes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}