# How OpenSEO Performs Site Audits: A Deep Dive into Its Durable Cloudflare Workflow

> OpenSEO performs site audits using a five-phase Cloudflare workflow. Discover how it crawls pages, runs Lighthouse checks, and generates comprehensive SEO reports.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-09-01

---

**OpenSEO performs site audits through a five-phase durable Cloudflare workflow that crawls pages, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.**

OpenSEO is an open-source SEO auditing tool built on Cloudflare's edge infrastructure. Understanding how OpenSEO performs site audits reveals a robust, retry-able pipeline designed for reliability at scale. This article examines the complete technical implementation based on the every-app/open-seo source code.

## The Five-Phase Audit Architecture

The `SiteAuditWorkflow` class in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) serves as the entry point. It validates request context and delegates to `runAuditPhases`, which executes five discrete phases. Each phase is wrapped in a **durable step** (`pgStep`) so failures can be retried without restarting the entire audit.

| Phase | File | Key Function |
|-------|------|--------------|
| Discovery | [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) | `runDiscoveryPhase` |
| Crawl | [`siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowCrawl.ts) | `runCrawlPhase` |
| Lighthouse | [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) | `runLighthousePhase` |
| Multipage checks | [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) | `runMultipageChecks` |
| Finalisation | [`siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/siteAuditWorkflowPhases.ts) | `finalizeAudit` |

```ts
// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
  // ...
}

```

## Phase 1: Discovery — Building the Seed URL List

The `runDiscoveryPhase` function fetches [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt), extracts sitemap URLs, and constructs an initial list of crawlable pages.

```ts
// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

```

Key implementation details:

- **Sitemap seed capping**: URLs are limited by `SITEMAP_SEED_BYTE_BUDGET` to stay under Cloudflare's 1 MiB step output limit
- **Progress persistence**: The audit status is immediately updated to `"crawling"` with the estimated page total

## Phase 2: Crawl — Breadth-First Page Collection

The `runCrawlPhase` in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) implements a **BFS crawl algorithm** with strict filtering rules. Only URLs satisfying all conditions are processed:

- Same-origin via `isSameOrigin`
- Crawlable via `isCrawlableUrl`
- Allowed by robots.txt via `robots.isAllowed`
- Not already visited or queued

The crawl maintains four in-memory collections for efficient frontier management:

| Collection | Purpose |
|------------|---------|
| `visited` | Deduplication: URLs already processed |
| `queued` | Batch staging: URLs added but not yet crawled |
| `linkQueue` | BFS queue for links discovered during crawling |
| `sitemapQueue` | BFS queue for sitemap-only URLs (drained last) |

```ts
// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ /* ... */ });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  // Slim summaries in memory; full data persisted to D1
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ /* ... */ });
  await persistCrawlProgress({ /* ... */ });
}

```

### Page-Level Crawling with `crawlPage`

The `crawlPage` function in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) handles actual HTTP requests with SEO-specific processing:

```ts
const response = await fetch(url, {
  headers: { 
    "User-Agent": CRAWL_USER_AGENT, 
    Accept: "text/html,application/xhtml+xml" 
  },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
// ...
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  fetchClass,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  // ... OG tags, headings, word count, image alt stats
  responseTimeMs,
  crawlDepth,
  inSitemap,
};

```

Notable `crawlPage` behaviors:

- **Custom User-Agent**: `OpenSEO-Audit/1.0` identifies the crawler
- **Manual redirects**: Each hop recorded separately for full redirect chain analysis
- **Bot challenge detection**: Recognizes 401/403/429/503 status codes and `CHALLENGE_BODY_MARKERS`
- **HTML size limit**: 2 MiB cap keeps workers cost-efficient
- **Deferred parsing**: `analyzeHtml` (Cheerio) imported dynamically only for valid HTML responses

## Phase 3: Lighthouse — Performance & Core Web Vitals

When `config.lighthouseStrategy !== "none"`, `runLighthousePhase` runs mobile and desktop Lighthouse reports on a sample or all crawled pages. Batching by `LIGHTHOUSE_URL_BATCH_SIZE` (10 URLs) prevents timeout issues.

```ts
// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, /* ... */);
  // Triggers fetchAndStoreLighthouseResult for mobile + desktop per URL
}

```

Results are persisted via `AuditRepository.insertLighthouseResults` and progress updated incrementally.

## Phase 4: Multipage Checks — Cross-Page SEO Analysis

After crawling completes, `runMultipageChecks` identifies site-wide issues:

- Orphan pages (no internal links)
- Duplicate titles
- Duplicate meta descriptions
- Missing hreflang annotations
- Canonical inconsistencies

```ts
const issues = await runMultipageChecks({ 
  auditId, 
  startUrl, 
  crawlCompleted: crawl.completed 
});
await AuditRepository.insertIssues(auditId, issues);

```

## Phase 5: Finalisation — Persistence & Telemetry

The `finalizeAudit` function completes the workflow:

```ts
await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", /* ... */ });
await AuditProgressKV.clear(auditId);

```

Error handling ensures failed audits report to PostHog, mark the audit as failed, and re-throw for durable object failure recording.

## Durability & Persistence Mechanisms

Three layers ensure audit reliability:

1. **Durable steps (`pgStep`)**: Cloudflare Durable Object steps enable per-phase retry without re-execution
2. **Database abstraction**: `AuditRepository` works with both SQLite (D1) and PostgreSQL
3. **KV progress store**: `AuditProgressKV` maintains crawl frontier state for fast status polling

## Starting an Audit: Practical Usage

```ts
import { AuditService } from "@/server/features/audit/services/AuditService";

// Initiate audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // "none" | "full" | "sample"
  },
  billingCustomer: { userId: "user_456", organizationId: "org_789" },
});

// Poll status
const status = await AuditService.getStatus(auditId, "proj_123");
// Returns: { phase: "crawling" | "lighthouse" | "completed" | "failed", pagesCrawled: number, ... }

// Retrieve results when complete
const results = await AuditService.getResults(auditId, "proj_123");

```

## Summary

- **OpenSEO site audits** run as durable Cloudflare workflows with five distinct, retry-able phases
- **Discovery** fetches robots.txt and sitemaps, capping seeds to 1 MiB output limits
- **Crawling** uses BFS with same-origin, robots.txt, and deduplication rules; `crawlPage` handles 15s timeouts, manual redirects, and bot detection
- **Lighthouse** runs mobile + desktop reports in batches of 10 URLs
- **Multipage checks** detect cross-page SEO issues like orphans and duplicates
- **Finalisation** persists results, emits telemetry, and cleans temporary state

## Frequently Asked Questions

### How does OpenSEO handle websites with thousands of pages?

OpenSEO respects the `maxPages` configuration parameter (defaulting based on plan). The crawl phase stops when this limit is reached, prioritizing sitemap seeds first, then BFS-discovered links. The `SITEMAP_SEED_BYTE_BUDGET` ensures the discovery phase output stays within Cloudflare's 1 MiB step limit.

### What happens if a page blocks the OpenSEO crawler?

The `crawlPage` function classifies responses using `fetchClass` based on status codes (401/403/429/503) and body content matching `CHALLENGE_BODY_MARKERS`. Blocked pages are recorded with their classification but excluded from HTML analysis, allowing the audit to continue with available data.

### Can OpenSEO audits resume after a failure?

Yes. Because each phase wraps operations in `pgStep` durable steps, failures restart at the failed step rather than the beginning. The `AuditProgressKV` store maintains frontier state, and `AuditRepository` persists completed page data, enabling incremental progress across retries.

### How do I configure Lighthouse sampling versus full-site analysis?

Set `lighthouseStrategy` to `"sample"` (representative subset), `"full"` (all pages), or `"none"` (skip). The `runLighthousePhase` automatically adjusts batch processing based on this configuration, with `"sample"` using statistical page selection to balance coverage and execution time.