How OpenSEO Performs Site Audits: Inside the Cloudflare Durable Workflow

OpenSEO executes site audits through a five-phase Cloudflare durable workflow that discovers URLs, performs a breadth-first crawl, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.

OpenSEO is an open-source SEO platform that automates technical site analysis using Cloudflare's durable workflow infrastructure. The audit system implemented in the every-app/open-seo repository splits the crawling process into discrete, retryable phases to ensure resilience against failures. Understanding how these site audits are performed reveals a sophisticated pipeline designed for accuracy and scale.

The Site Audit Entry Point

Every audit begins in src/server/workflows/SiteAuditWorkflow.ts with the SiteAuditWorkflow class. This entry point validates the request context, then delegates to runAuditPhases to execute the pipeline.

// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
  // ...
}

The workflow wraps execution in withPgClient to ensure database connections are properly scoped. If any phase throws an error, the workflow captures the failure, reports it to PostHog, marks the audit as failed, and re-throws to let the durable object record the termination.

Phase 1: Discovery

The runDiscoveryPhase function in src/server/workflows/siteAuditWorkflowPhases.ts initiates the audit by fetching robots.txt and extracting sitemap URLs to build a seed list.

// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

The discoverUrls utility fetches the site's robots.txt and any linked XML sitemaps. To stay under Cloudflare's 1 MiB step output limit, the system enforces a SITEMAP_SEED_BYTE_BUDGET when capping sitemap seeds. After discovery, the audit progress is updated with the total page count and the phase is marked as crawling.

Phase 2: Breadth-First Crawl

The runCrawlPhase in src/server/workflows/siteAuditWorkflowCrawl.ts implements a BFS crawl that respects same-origin policy, robots.txt directives, and crawlability rules. It maintains three in-memory collections to manage state:

• visited – URLs already processed for deduplication
• queued – URLs added to a batch but not yet processed
• linkQueue / sitemapQueue – Separate BFS queues for discovered links versus sitemap-only URLs

// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ … });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ … });
  await persistCrawlProgress({ … });
}

Page-Level Crawling Logic

The crawlPage function in src/server/workflows/site-audit-workflow-helpers.ts performs the actual HTTP request using a custom User-Agent: OpenSEO-Audit/1.0. It handles redirects manually (recording each hop separately), detects bot-mitigation challenges via status codes (401/403/429/503) and body markers (CHALLENGE_BODY_MARKERS), and limits HTML size to 2 MiB to control worker costs.

// src/server/workflows/site-audit-workflow-helpers.ts
const response = await fetch(url, {
  headers: { "User-Agent": CRAWL_USER_AGENT, Accept: "text/html,application/xhtml+xml" },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
// ...
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  fetchClass,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  canonicalUrl: analysis.canonicalUrl,
  ogTags: analysis.ogTags,
  headingCounts: analysis.headingCounts,
  wordCount: analysis.wordCount,
  imageAltTextStats: analysis.imageAltTextStats,
  responseTimeMs,
  crawlDepth,
  inSitemap,
};

The analyzeHtml function (dynamically imported Cheerio) extracts SEO-critical data including title tags, meta descriptions, canonical links, Open Graph tags, heading hierarchies, and image alt-text statistics. Full page data is persisted to the database via AuditRepository.insertCrawledBatch, while only slim summaries are retained in the workflow heap.

Phase 3: Lighthouse Performance Audits

If the audit configuration specifies a lighthouseStrategy other than "none", the runLighthousePhase selects a representative sample (or all pages) and generates mobile and desktop Lighthouse reports.

// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, …);
  // ...
}

The system processes URLs in batches of LIGHTHOUSE_URL_BATCH_SIZE (10 URLs) to manage resource consumption. Each URL triggers two fetchAndStoreLighthouseResult calls—one for mobile and one for desktop—which store results via AuditRepository.insertLighthouseResults.

Phase 4: Multipage Checks and Finalization

After crawling completes, runMultipageChecks executes cross-page SEO validation in src/server/workflows/siteAuditWorkflowPhases.ts. This detects orphan pages, duplicate titles, missing hreflang annotations, and other site-wide issues.

// src/server/workflows/siteAuditWorkflowPhases.ts
const issues = await runMultipageChecks({ auditId, startUrl, crawlCompleted: crawl.completed });
await AuditRepository.insertIssues(auditId, issues);

Finally, finalizeAudit marks the audit as complete, records telemetry to PostHog, and clears temporary progress storage:

await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", … });
await AuditProgressKV.clear(auditId);

Durability and Persistence Architecture

OpenSEO's audit workflow leverages Cloudflare Durable Objects to ensure reliability:

• pgStep wrapper – Every logical step is wrapped in a durable step that can be retried without re-executing previous work
• Database abstraction – AuditRepository works with both SQLite (D1) and Postgres, persisting crawled pages, Lighthouse results, and discovered issues
• KV progress store – AuditProgressKV maintains intermediate crawl state (frontier counts, visited size) for real-time status queries without hitting the primary database

How to Trigger a Site Audit Programmatically

To start an audit from your application, use the AuditService via the server functions API:

import { AuditService } from "@/server/features/audit/services/AuditService";

// Initiate the audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // "none" | "full" | "sample"
  },
  billingCustomer: {
    userId: "user_456",
    organizationId: "org_789",
  },
});

// Poll for status
const status = await AuditService.getStatus(auditId, "proj_123");
if (status.phase === "completed") {
  const results = await AuditService.getResults(auditId, "proj_123");
  console.log("Issues found:", results.issues.length);
}

The startAudit function validates permissions, creates the audit record, and triggers the durable workflow. Status updates are available through getStatus, which reads from the progress KV store during active crawling and the database upon completion.

Summary

• Five-phase pipeline – Discovery, Crawl, Lighthouse, Multipage Checks, and Finalization execute sequentially in isolated, retryable steps
• BFS crawling – Respects same-origin policy, robots.txt directives, and implements bot-challenge detection with a custom user agent
• Resource limits – Enforces 2 MiB HTML size limits, 15-second timeouts, and configurable maxPages constraints to control worker costs
• Durable execution – Cloudflare Workflow steps ensure that failures in later phases do not require restarting the entire audit
• Comprehensive extraction – Captures titles, meta descriptions, canonicals, Open Graph tags, heading structures, and image accessibility data
• Flexible Lighthouse – Supports full-site, sampled, or disabled performance auditing with separate mobile and desktop reports

Frequently Asked Questions

How does OpenSEO handle robots.txt during site audits?

During the Discovery phase, discoverUrls fetches and parses the target site's robots.txt file. The subsequent Crawl phase uses these rules to filter URLs via robots.isAllowed checks before adding them to the crawl queue. The system also respects same-origin policies and crawlability heuristics defined in isCrawlableUrl.

What happens when OpenSEO encounters a bot challenge or block?

The crawlPage function classifies responses using status codes (401, 403, 429, 503) and body content markers (CHALLENGE_BODY_MARKERS). When detected, the page is recorded with a fetchClass indicating the block type rather than attempting to parse the HTML, allowing the audit to continue without hanging on protected pages.

Can OpenSEO run Lighthouse on every page or just a sample?

The lighthouseStrategy configuration option supports three modes: "none" skips Lighthouse entirely, "sample" audits a representative subset of pages, and "full" runs performance checks on every crawled URL. Regardless of mode, Lighthouse execution batches URLs into groups of 10 (LIGHTHOUSE_URL_BATCH_SIZE) to manage resource consumption.

How is crawl data stored during the audit process?

Crawled page data is persisted to the configured database (D1 or Postgres) via AuditRepository.insertCrawledBatch after each batch completes. Intermediate progress metrics (pages crawled, current phase) are stored in AuditProgressKV for low-latency status polling. Once the audit completes, temporary KV entries are cleared while the persistent audit record remains queryable through AuditService.getResults.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}