How OpenSEO Performs Site Audits: A Technical Breakdown of the Cloudflare Workflow

OpenSEO executes site audits through a durable Cloudflare workflow that discovers URLs via robots.txt and sitemaps, performs a breadth-first crawl respecting same-origin and robots.txt rules, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.

OpenSEO is an open-source SEO auditing platform that leverages Cloudflare's durable workflows to crawl websites and generate detailed performance reports. Understanding how OpenSEO performs site audits reveals a sophisticated pipeline designed for resilience, handling complex sites through discrete, retryable phases that prevent total workflow restarts when individual steps encounter errors.

Workflow Architecture and Entry Point

The audit process begins with the SiteAuditWorkflow class in src/server/workflows/SiteAuditWorkflow.ts, which serves as the Cloudflare Workflow entry point. This class validates the request context and orchestrates the entire pipeline through the runAuditPhases function.

// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
}

The workflow executes five distinct phases: Discovery, Crawl, Lighthouse, Multipage Checks, and Finalisation. Each phase is wrapped in a durable step using pgStep, enabling automatic retries without re-executing previous successful work.

Phase 1: Discovery and Seed URL Collection

The runDiscoveryPhase function in src/server/workflows/siteAuditWorkflowPhases.ts initiates the audit by fetching the target site's robots.txt and extracting sitemap URLs. This phase builds the initial seed list for crawling while respecting the 1 MiB step output limit defined by SITEMAP_SEED_BYTE_BUDGET.

// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

The phase caps sitemap seeds to prevent memory overflow and immediately persists the projected page count to the database, updating the audit status to "crawling" for real-time progress tracking.

Phase 2: Breadth-First Crawl Implementation

The crawl phase operates as a breadth-first search (BFS) algorithm implemented in src/server/workflows/siteAuditWorkflowCrawl.ts. The runCrawlPhase function maintains strict crawling rules: same-origin URLs only, robots.txt compliance, and deduplication through three in-memory collections:

• visited: URLs already processed
• queued: URLs added to active batches
• linkQueue / sitemapQueue: Separate BFS queues for discovered links versus sitemap-only URLs

// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ … });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ … });
  await persistCrawlProgress({ … });
}

Low-Level Page Crawling

Individual page fetching occurs in crawlPage within src/server/workflows/site-audit-workflow-helpers.ts. This function uses a custom User-Agent: OpenSEO-Audit/1.0, enforces a 15-second timeout, and limits HTML size to 2 MiB to control worker costs.

// src/server/workflows/site-audit-workflow-helpers.ts
const response = await fetch(url, {
  headers: { "User-Agent": CRAWL_USER_AGENT, Accept: "text/html,application/xhtml+xml" },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  canonicalLink: analysis.canonicalLink,
  ogTags: analysis.ogTags,
  headingCounts: analysis.headingCounts,
  wordCount: analysis.wordCount,
  responseTimeMs,
};

The function handles manual redirects (recording each hop separately), detects bot mitigation blocks via status codes (401/403/429/503) and body markers (CHALLENGE_BODY_MARKERS), and defers HTML parsing to Cheerio only when processing valid HTML responses.

Phase 3: Lighthouse Performance Testing

When config.lighthouseStrategy is not set to "none", the runLighthousePhase function selects a representative sample (or all pages) and executes mobile and desktop Lighthouse reports in batches of 10 URLs (LIGHTHOUSE_URL_BATCH_SIZE).

// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, …);
  // Triggers fetchAndStoreLighthouseResult for each URL × 2 devices
}

Results are persisted via AuditRepository.insertLighthouseResults, updating the audit progress after each batch completion.

Phase 4: Cross-Page Analysis and Finalization

After crawling completes, runMultipageChecks in src/server/workflows/siteAuditWorkflowPhases.ts executes cross-page SEO rules to detect orphan pages, duplicate titles, and missing hreflang attributes. Issues are batch-inserted via AuditRepository.insertIssues.

// src/server/workflows/siteAuditWorkflowPhases.ts
const issues = await runMultipageChecks({ auditId, startUrl, crawlCompleted: crawl.completed });
await AuditRepository.insertIssues(auditId, issues);

The finalizeAudit function marks the audit complete, transmits telemetry to PostHog, and clears temporary KV storage:

await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", … });
await AuditProgressKV.clear(auditId);

If any phase throws an exception, the workflow captures the error, reports it to PostHog, marks the audit as failed, and re-throws to ensure the durable object records the failure state.

Durability and Persistence Mechanisms

OpenSEO achieves fault tolerance through Cloudflare Durable Objects wrapped in pgStep calls, allowing each phase to retry independently. The architecture supports multiple database backends:

• Postgres/D1: Stores crawled pages, Lighthouse results, and SEO issues via AuditRepository
• KV Storage: Maintains crawl frontier state (visited counts, queue sizes) in AuditProgressKV for low-latency status polling
• Error Isolation: Failures in Lighthouse or multipage checks do not invalidate successfully crawled data

Practical Usage: Triggering an Audit

To initiate an OpenSEO site audit programmatically, use the AuditService.startAudit method exposed through src/serverFunctions/audit.ts:

import { AuditService } from "@/server/features/audit/services/AuditService";

// Start audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // Options: "none" | "sample" | "full"
  },
  billingCustomer: {
    userId: "user_456",
    organizationId: "org_789",
  },
});

// Poll for progress
const status = await AuditService.getStatus(auditId, "proj_123");
// Returns: { phase: "crawling" | "lighthouse" | "completed", pagesCrawled: number, ... }

// Retrieve results when complete
const results = await AuditService.getResults(auditId, "proj_123");

Summary

• OpenSEO performs site audits using a durable Cloudflare workflow split into five discrete, retryable phases.
• The Discovery phase parses robots.txt and sitemaps to build seed URLs while respecting byte budget limits.
• The Crawl phase implements BFS with strict same-origin and robots.txt validation, using a custom user agent and 2 MiB HTML size limits.
• Lighthouse testing runs in batches of 10 URLs across mobile and desktop devices when enabled.
• Multipage checks detect cross-site SEO issues like duplicate titles and orphan pages before finalization.
• All data persists to D1 or Postgres via AuditRepository, with intermediate state stored in KV for real-time progress updates.

Frequently Asked Questions

How does OpenSEO respect robots.txt directives?

According to the source code in src/server/workflows/siteAuditWorkflowCrawl.ts, the crawler validates every URL against the parsed robots.txt rules using robots.isAllowed before adding it to the crawl queue. The Discovery phase first fetches and parses robots.txt in src/server/workflows/siteAuditWorkflowPhases.ts, making those rules available to the BFS crawl logic that filters URLs through isCrawlableUrl checks.

What happens when OpenSEO encounters bot protection or crawling errors?

The crawlPage function in src/server/workflows/site-audit-workflow-helpers.ts detects bot challenges through HTTP status codes (401, 403, 429, 503) and body content markers defined in CHALLENGE_BODY_MARKERS. When blocked, the crawler records the specific status code and classification without extracting HTML content, allowing the audit to continue with other URLs while preserving error metadata for the final report.

How does the workflow handle partial failures during long audits?

OpenSEO wraps each logical phase in pgStep calls (visible in src/server/workflows/siteAuditWorkflowPhases.ts), which leverage Cloudflare Durable Objects to create durable execution steps. If a specific step fails—for example, a Lighthouse batch timing out—the workflow retries that specific step without re-executing previously completed crawl work. The error handling in SiteAuditWorkflow.ts captures exceptions, reports them to PostHog, marks the audit as failed in the database, and re-throws to maintain accurate state tracking.

What is the maximum crawl size OpenSEO supports?

The maxPages configuration parameter controls crawl limits, with the system designed to respect the 1 MiB step output limit (SITEMAP_SEED_BYTE_BUDGET) for sitemap seeds and 2 MiB HTML size limits per page. The BFS algorithm in src/server/workflows/siteAuditWorkflowCrawl.ts continuously checks summaries.length < maxPages to enforce these boundaries, while batch processing via runCrawlBatch ensures memory remains bounded even for large-scale audits.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}