How OpenSEO Performs Site Audits: A Deep Dive into Its Durable Cloudflare Workflow

OpenSEO performs site audits through a five-phase durable Cloudflare workflow that crawls pages, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.

OpenSEO is an open-source SEO auditing tool built on Cloudflare's edge infrastructure. Understanding how OpenSEO performs site audits reveals a robust, retry-able pipeline designed for reliability at scale. This article examines the complete technical implementation based on the every-app/open-seo source code.

The Five-Phase Audit Architecture

The SiteAuditWorkflow class in src/server/workflows/SiteAuditWorkflow.ts serves as the entry point. It validates request context and delegates to runAuditPhases, which executes five discrete phases. Each phase is wrapped in a durable step (pgStep) so failures can be retried without restarting the entire audit.

Phase	File	Key Function
Discovery	siteAuditWorkflowPhases.ts	runDiscoveryPhase
Crawl	siteAuditWorkflowCrawl.ts	runCrawlPhase
Lighthouse	siteAuditWorkflowPhases.ts	runLighthousePhase
Multipage checks	siteAuditWorkflowPhases.ts	runMultipageChecks
Finalisation	siteAuditWorkflowPhases.ts	finalizeAudit

// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
  // ...
}

Phase 1: Discovery — Building the Seed URL List

The runDiscoveryPhase function fetches robots.txt, extracts sitemap URLs, and constructs an initial list of crawlable pages.

// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

Key implementation details:

• Sitemap seed capping: URLs are limited by SITEMAP_SEED_BYTE_BUDGET to stay under Cloudflare's 1 MiB step output limit
• Progress persistence: The audit status is immediately updated to "crawling" with the estimated page total

Phase 2: Crawl — Breadth-First Page Collection

The runCrawlPhase in src/server/workflows/siteAuditWorkflowCrawl.ts implements a BFS crawl algorithm with strict filtering rules. Only URLs satisfying all conditions are processed:

• Same-origin via isSameOrigin
• Crawlable via isCrawlableUrl
• Allowed by robots.txt via robots.isAllowed
• Not already visited or queued

The crawl maintains four in-memory collections for efficient frontier management:

Collection	Purpose
visited	Deduplication: URLs already processed
queued	Batch staging: URLs added but not yet crawled
linkQueue	BFS queue for links discovered during crawling
sitemapQueue	BFS queue for sitemap-only URLs (drained last)

// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ /* ... */ });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  // Slim summaries in memory; full data persisted to D1
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ /* ... */ });
  await persistCrawlProgress({ /* ... */ });
}

Page-Level Crawling with crawlPage

The crawlPage function in src/server/workflows/site-audit-workflow-helpers.ts handles actual HTTP requests with SEO-specific processing:

const response = await fetch(url, {
  headers: { 
    "User-Agent": CRAWL_USER_AGENT, 
    Accept: "text/html,application/xhtml+xml" 
  },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
// ...
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  fetchClass,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  // ... OG tags, headings, word count, image alt stats
  responseTimeMs,
  crawlDepth,
  inSitemap,
};

Notable crawlPage behaviors:

• Custom User-Agent: OpenSEO-Audit/1.0 identifies the crawler
• Manual redirects: Each hop recorded separately for full redirect chain analysis
• Bot challenge detection: Recognizes 401/403/429/503 status codes and CHALLENGE_BODY_MARKERS
• HTML size limit: 2 MiB cap keeps workers cost-efficient
• Deferred parsing: analyzeHtml (Cheerio) imported dynamically only for valid HTML responses

Phase 3: Lighthouse — Performance & Core Web Vitals

When config.lighthouseStrategy !== "none", runLighthousePhase runs mobile and desktop Lighthouse reports on a sample or all crawled pages. Batching by LIGHTHOUSE_URL_BATCH_SIZE (10 URLs) prevents timeout issues.

// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, /* ... */);
  // Triggers fetchAndStoreLighthouseResult for mobile + desktop per URL
}

Results are persisted via AuditRepository.insertLighthouseResults and progress updated incrementally.

Phase 4: Multipage Checks — Cross-Page SEO Analysis

After crawling completes, runMultipageChecks identifies site-wide issues:

• Orphan pages (no internal links)
• Duplicate titles
• Duplicate meta descriptions
• Missing hreflang annotations
• Canonical inconsistencies

const issues = await runMultipageChecks({ 
  auditId, 
  startUrl, 
  crawlCompleted: crawl.completed 
});
await AuditRepository.insertIssues(auditId, issues);

Phase 5: Finalisation — Persistence & Telemetry

The finalizeAudit function completes the workflow:

await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", /* ... */ });
await AuditProgressKV.clear(auditId);

Error handling ensures failed audits report to PostHog, mark the audit as failed, and re-throw for durable object failure recording.

Durability & Persistence Mechanisms

Three layers ensure audit reliability:

1. Durable steps (pgStep): Cloudflare Durable Object steps enable per-phase retry without re-execution
2. Database abstraction: AuditRepository works with both SQLite (D1) and PostgreSQL
3. KV progress store: AuditProgressKV maintains crawl frontier state for fast status polling

Starting an Audit: Practical Usage

import { AuditService } from "@/server/features/audit/services/AuditService";

// Initiate audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // "none" | "full" | "sample"
  },
  billingCustomer: { userId: "user_456", organizationId: "org_789" },
});

// Poll status
const status = await AuditService.getStatus(auditId, "proj_123");
// Returns: { phase: "crawling" | "lighthouse" | "completed" | "failed", pagesCrawled: number, ... }

// Retrieve results when complete
const results = await AuditService.getResults(auditId, "proj_123");

Summary

• OpenSEO site audits run as durable Cloudflare workflows with five distinct, retry-able phases
• Discovery fetches robots.txt and sitemaps, capping seeds to 1 MiB output limits
• Crawling uses BFS with same-origin, robots.txt, and deduplication rules; crawlPage handles 15s timeouts, manual redirects, and bot detection
• Lighthouse runs mobile + desktop reports in batches of 10 URLs
• Multipage checks detect cross-page SEO issues like orphans and duplicates
• Finalisation persists results, emits telemetry, and cleans temporary state

Frequently Asked Questions

How does OpenSEO handle websites with thousands of pages?

OpenSEO respects the maxPages configuration parameter (defaulting based on plan). The crawl phase stops when this limit is reached, prioritizing sitemap seeds first, then BFS-discovered links. The SITEMAP_SEED_BYTE_BUDGET ensures the discovery phase output stays within Cloudflare's 1 MiB step limit.

What happens if a page blocks the OpenSEO crawler?

The crawlPage function classifies responses using fetchClass based on status codes (401/403/429/503) and body content matching CHALLENGE_BODY_MARKERS. Blocked pages are recorded with their classification but excluded from HTML analysis, allowing the audit to continue with available data.

Can OpenSEO audits resume after a failure?

Yes. Because each phase wraps operations in pgStep durable steps, failures restart at the failed step rather than the beginning. The AuditProgressKV store maintains frontier state, and AuditRepository persists completed page data, enabling incremental progress across retries.

How do I configure Lighthouse sampling versus full-site analysis?

Set lighthouseStrategy to "sample" (representative subset), "full" (all pages), or "none" (skip). The runLighthousePhase automatically adjusts batch processing based on this configuration, with "sample" using statistical page selection to balance coverage and execution time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}