# How OpenSEO Performs Site Audits: Inside the Cloudflare Durable Workflow

> Discover how OpenSEO performs site audits using a five-phase Cloudflare durable workflow. Get comprehensive SEO reports with URL discovery, breadth-first crawling, and Lighthouse checks.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-30

---

**OpenSEO executes site audits through a five-phase Cloudflare durable workflow that discovers URLs, performs a breadth-first crawl, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.**

OpenSEO is an open-source SEO platform that automates technical site analysis using Cloudflare's durable workflow infrastructure. The audit system implemented in the `every-app/open-seo` repository splits the crawling process into discrete, retryable phases to ensure resilience against failures. Understanding how these **site audits** are performed reveals a sophisticated pipeline designed for accuracy and scale.

## The Site Audit Entry Point

Every audit begins in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts) with the `SiteAuditWorkflow` class. This entry point validates the request context, then delegates to `runAuditPhases` to execute the pipeline.

```typescript
// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
  // ...
}

```

The workflow wraps execution in `withPgClient` to ensure database connections are properly scoped. If any phase throws an error, the workflow captures the failure, reports it to PostHog, marks the audit as failed, and re-throws to let the durable object record the termination.

## Phase 1: Discovery

The `runDiscoveryPhase` function in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) initiates the audit by fetching **robots.txt** and extracting sitemap URLs to build a seed list.

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

```

The `discoverUrls` utility fetches the site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) and any linked XML sitemaps. To stay under Cloudflare's 1 MiB step output limit, the system enforces a `SITEMAP_SEED_BYTE_BUDGET` when capping sitemap seeds. After discovery, the audit progress is updated with the total page count and the phase is marked as **crawling**.

## Phase 2: Breadth-First Crawl

The `runCrawlPhase` in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) implements a **BFS crawl** that respects same-origin policy, robots.txt directives, and crawlability rules. It maintains three in-memory collections to manage state:

- **`visited`** – URLs already processed for deduplication
- **`queued`** – URLs added to a batch but not yet processed
- **`linkQueue`** / **`sitemapQueue`** – Separate BFS queues for discovered links versus sitemap-only URLs

```typescript
// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ … });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ … });
  await persistCrawlProgress({ … });
}

```

### Page-Level Crawling Logic

The `crawlPage` function in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) performs the actual HTTP request using a custom `User-Agent: OpenSEO-Audit/1.0`. It handles redirects manually (recording each hop separately), detects bot-mitigation challenges via status codes (401/403/429/503) and body markers (`CHALLENGE_BODY_MARKERS`), and limits HTML size to **2 MiB** to control worker costs.

```typescript
// src/server/workflows/site-audit-workflow-helpers.ts
const response = await fetch(url, {
  headers: { "User-Agent": CRAWL_USER_AGENT, Accept: "text/html,application/xhtml+xml" },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
// ...
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  fetchClass,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  canonicalUrl: analysis.canonicalUrl,
  ogTags: analysis.ogTags,
  headingCounts: analysis.headingCounts,
  wordCount: analysis.wordCount,
  imageAltTextStats: analysis.imageAltTextStats,
  responseTimeMs,
  crawlDepth,
  inSitemap,
};

```

The `analyzeHtml` function (dynamically imported Cheerio) extracts SEO-critical data including title tags, meta descriptions, canonical links, Open Graph tags, heading hierarchies, and image alt-text statistics. Full page data is persisted to the database via `AuditRepository.insertCrawledBatch`, while only slim summaries are retained in the workflow heap.

## Phase 3: Lighthouse Performance Audits

If the audit configuration specifies a `lighthouseStrategy` other than `"none"`, the `runLighthousePhase` selects a representative sample (or all pages) and generates mobile and desktop Lighthouse reports.

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, …);
  // ...
}

```

The system processes URLs in batches of `LIGHTHOUSE_URL_BATCH_SIZE` (10 URLs) to manage resource consumption. Each URL triggers two `fetchAndStoreLighthouseResult` calls—one for mobile and one for desktop—which store results via `AuditRepository.insertLighthouseResults`.

## Phase 4: Multipage Checks and Finalization

After crawling completes, `runMultipageChecks` executes cross-page SEO validation in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts). This detects **orphan pages**, **duplicate titles**, **missing hreflang** annotations, and other site-wide issues.

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
const issues = await runMultipageChecks({ auditId, startUrl, crawlCompleted: crawl.completed });
await AuditRepository.insertIssues(auditId, issues);

```

Finally, `finalizeAudit` marks the audit as complete, records telemetry to PostHog, and clears temporary progress storage:

```typescript
await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", … });
await AuditProgressKV.clear(auditId);

```

## Durability and Persistence Architecture

OpenSEO's audit workflow leverages **Cloudflare Durable Objects** to ensure reliability:

- **`pgStep` wrapper** – Every logical step is wrapped in a durable step that can be retried without re-executing previous work
- **Database abstraction** – `AuditRepository` works with both SQLite (D1) and Postgres, persisting crawled pages, Lighthouse results, and discovered issues
- **KV progress store** – `AuditProgressKV` maintains intermediate crawl state (frontier counts, visited size) for real-time status queries without hitting the primary database

## How to Trigger a Site Audit Programmatically

To start an audit from your application, use the `AuditService` via the server functions API:

```typescript
import { AuditService } from "@/server/features/audit/services/AuditService";

// Initiate the audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // "none" | "full" | "sample"
  },
  billingCustomer: {
    userId: "user_456",
    organizationId: "org_789",
  },
});

// Poll for status
const status = await AuditService.getStatus(auditId, "proj_123");
if (status.phase === "completed") {
  const results = await AuditService.getResults(auditId, "proj_123");
  console.log("Issues found:", results.issues.length);
}

```

The `startAudit` function validates permissions, creates the audit record, and triggers the durable workflow. Status updates are available through `getStatus`, which reads from the progress KV store during active crawling and the database upon completion.

## Summary

- **Five-phase pipeline** – Discovery, Crawl, Lighthouse, Multipage Checks, and Finalization execute sequentially in isolated, retryable steps
- **BFS crawling** – Respects same-origin policy, robots.txt directives, and implements bot-challenge detection with a custom user agent
- **Resource limits** – Enforces 2 MiB HTML size limits, 15-second timeouts, and configurable `maxPages` constraints to control worker costs
- **Durable execution** – Cloudflare Workflow steps ensure that failures in later phases do not require restarting the entire audit
- **Comprehensive extraction** – Captures titles, meta descriptions, canonicals, Open Graph tags, heading structures, and image accessibility data
- **Flexible Lighthouse** – Supports full-site, sampled, or disabled performance auditing with separate mobile and desktop reports

## Frequently Asked Questions

### How does OpenSEO handle robots.txt during site audits?

During the Discovery phase, `discoverUrls` fetches and parses the target site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) file. The subsequent Crawl phase uses these rules to filter URLs via `robots.isAllowed` checks before adding them to the crawl queue. The system also respects same-origin policies and crawlability heuristics defined in `isCrawlableUrl`.

### What happens when OpenSEO encounters a bot challenge or block?

The `crawlPage` function classifies responses using status codes (401, 403, 429, 503) and body content markers (`CHALLENGE_BODY_MARKERS`). When detected, the page is recorded with a `fetchClass` indicating the block type rather than attempting to parse the HTML, allowing the audit to continue without hanging on protected pages.

### Can OpenSEO run Lighthouse on every page or just a sample?

The `lighthouseStrategy` configuration option supports three modes: `"none"` skips Lighthouse entirely, `"sample"` audits a representative subset of pages, and `"full"` runs performance checks on every crawled URL. Regardless of mode, Lighthouse execution batches URLs into groups of 10 (`LIGHTHOUSE_URL_BATCH_SIZE`) to manage resource consumption.

### How is crawl data stored during the audit process?

Crawled page data is persisted to the configured database (D1 or Postgres) via `AuditRepository.insertCrawledBatch` after each batch completes. Intermediate progress metrics (pages crawled, current phase) are stored in `AuditProgressKV` for low-latency status polling. Once the audit completes, temporary KV entries are cleared while the persistent audit record remains queryable through `AuditService.getResults`.