# How OpenSEO Performs Site Audits: A Technical Breakdown of the Cloudflare Workflow

> Discover how OpenSEO performs site audits using a Cloudflare workflow. This technical breakdown covers URL discovery, breadth-first crawling, Lighthouse checks, and comprehensive SEO issue aggregation for actionable insights.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: deep-dive
- Published: 2026-08-05

---

**OpenSEO executes site audits through a durable Cloudflare workflow that discovers URLs via robots.txt and sitemaps, performs a breadth-first crawl respecting same-origin and robots.txt rules, runs Lighthouse performance checks, and aggregates cross-page SEO issues into a comprehensive report.**

OpenSEO is an open-source SEO auditing platform that leverages Cloudflare's durable workflows to crawl websites and generate detailed performance reports. Understanding how OpenSEO performs site audits reveals a sophisticated pipeline designed for resilience, handling complex sites through discrete, retryable phases that prevent total workflow restarts when individual steps encounter errors.

## Workflow Architecture and Entry Point

The audit process begins with the `SiteAuditWorkflow` class in [`src/server/workflows/SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/SiteAuditWorkflow.ts), which serves as the Cloudflare Workflow entry point. This class validates the request context and orchestrates the entire pipeline through the `runAuditPhases` function.

```typescript
// src/server/workflows/SiteAuditWorkflow.ts
export class SiteAuditWorkflow extends WorkflowEntrypoint<Env, AuditParams> {
  async run(event: WorkflowEvent<AuditParams>, step: WorkflowStep) {
    return withPgClient(() => this.runScoped(event, step));
  }
}

```

The workflow executes five distinct phases: Discovery, Crawl, Lighthouse, Multipage Checks, and Finalisation. Each phase is wrapped in a durable step using `pgStep`, enabling automatic retries without re-executing previous successful work.

## Phase 1: Discovery and Seed URL Collection

The `runDiscoveryPhase` function in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) initiates the audit by fetching the target site's [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) and extracting sitemap URLs. This phase builds the initial seed list for crawling while respecting the 1 MiB step output limit defined by `SITEMAP_SEED_BYTE_BUDGET`.

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
async function runDiscoveryPhase(...) {
  return pgStep(step, "discover-urls", undefined, async () => {
    const result = await discoverUrls(origin, maxPages);
    await AuditRepository.updateAuditProgress(auditId, workflowInstanceId, {
      pagesTotal: Math.min(result.urls.length + 1, maxPages),
      currentPhase: "crawling",
    });
    return {
      sitemapUrls: capSitemapSeeds(result.urls, maxPages),
      robotsText: result.robotsText,
    };
  });
}

```

The phase caps sitemap seeds to prevent memory overflow and immediately persists the projected page count to the database, updating the audit status to "crawling" for real-time progress tracking.

## Phase 2: Breadth-First Crawl Implementation

The crawl phase operates as a **breadth-first search (BFS)** algorithm implemented in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts). The `runCrawlPhase` function maintains strict crawling rules: same-origin URLs only, [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) compliance, and deduplication through three in-memory collections:

- **visited**: URLs already processed
- **queued**: URLs added to active batches
- **linkQueue / sitemapQueue**: Separate BFS queues for discovered links versus sitemap-only URLs

```typescript
// src/server/workflows/siteAuditWorkflowCrawl.ts
while ((linkQueue.length > 0 || sitemapQueue.length > 0) &&
       summaries.length < maxPages) {
  const batchEntries = selectNextCrawlBatch({ … });
  crawlBatchIndex += 1;
  const crawledBatch = await runCrawlBatch(step, {
    crawlBatchIndex,
    auditId,
    batchEntries,
    sitemapSet,
    visited,
    queued,
  });
  summaries.push(...crawledBatch.map(({ internalLinks: _links, ...s }) => s));
  enqueueDiscoveredLinks({ … });
  await persistCrawlProgress({ … });
}

```

### Low-Level Page Crawling

Individual page fetching occurs in `crawlPage` within [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts). This function uses a custom `User-Agent: OpenSEO-Audit/1.0`, enforces a 15-second timeout, and limits HTML size to 2 MiB to control worker costs.

```typescript
// src/server/workflows/site-audit-workflow-helpers.ts
const response = await fetch(url, {
  headers: { "User-Agent": CRAWL_USER_AGENT, Accept: "text/html,application/xhtml+xml" },
  redirect: "manual",
  signal: AbortSignal.timeout(15_000),
});
const analysis = analyzeHtml(body, url, statusCode, responseTimeMs);
return {
  id: crypto.randomUUID(),
  url,
  statusCode,
  title: analysis.title,
  metaDescription: analysis.metaDescription,
  canonicalLink: analysis.canonicalLink,
  ogTags: analysis.ogTags,
  headingCounts: analysis.headingCounts,
  wordCount: analysis.wordCount,
  responseTimeMs,
};

```

The function handles manual redirects (recording each hop separately), detects bot mitigation blocks via status codes (401/403/429/503) and body markers (`CHALLENGE_BODY_MARKERS`), and defers HTML parsing to Cheerio only when processing valid HTML responses.

## Phase 3: Lighthouse Performance Testing

When `config.lighthouseStrategy` is not set to `"none"`, the `runLighthousePhase` function selects a representative sample (or all pages) and executes mobile and desktop Lighthouse reports in batches of 10 URLs (`LIGHTHOUSE_URL_BATCH_SIZE`).

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
for (let i = 0; i < lighthouseWork.length; i += LIGHTHOUSE_URL_BATCH_SIZE) {
  const batch = lighthouseWork.slice(i, i + LIGHTHOUSE_URL_BATCH_SIZE);
  const counts = await pgStep(step, `lighthouse-batch-${lighthouseBatchIndex}`, …);
  // Triggers fetchAndStoreLighthouseResult for each URL × 2 devices
}

```

Results are persisted via `AuditRepository.insertLighthouseResults`, updating the audit progress after each batch completion.

## Phase 4: Cross-Page Analysis and Finalization

After crawling completes, `runMultipageChecks` in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) executes cross-page SEO rules to detect orphan pages, duplicate titles, and missing hreflang attributes. Issues are batch-inserted via `AuditRepository.insertIssues`.

```typescript
// src/server/workflows/siteAuditWorkflowPhases.ts
const issues = await runMultipageChecks({ auditId, startUrl, crawlCompleted: crawl.completed });
await AuditRepository.insertIssues(auditId, issues);

```

The `finalizeAudit` function marks the audit complete, transmits telemetry to PostHog, and clears temporary KV storage:

```typescript
await AuditRepository.completeAudit(auditId, workflowInstanceId, {
  pagesCrawled: crawl.pages.length,
  pagesTotal: crawl.pages.length,
});
await captureServerEvent({ event: "site_audit:complete", … });
await AuditProgressKV.clear(auditId);

```

If any phase throws an exception, the workflow captures the error, reports it to PostHog, marks the audit as failed, and re-throws to ensure the durable object records the failure state.

## Durability and Persistence Mechanisms

OpenSEO achieves fault tolerance through **Cloudflare Durable Objects** wrapped in `pgStep` calls, allowing each phase to retry independently. The architecture supports multiple database backends:

- **Postgres/D1**: Stores crawled pages, Lighthouse results, and SEO issues via `AuditRepository`
- **KV Storage**: Maintains crawl frontier state (visited counts, queue sizes) in `AuditProgressKV` for low-latency status polling
- **Error Isolation**: Failures in Lighthouse or multipage checks do not invalidate successfully crawled data

## Practical Usage: Triggering an Audit

To initiate an OpenSEO site audit programmatically, use the `AuditService.startAudit` method exposed through [`src/serverFunctions/audit.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/audit.ts):

```typescript
import { AuditService } from "@/server/features/audit/services/AuditService";

// Start audit
const auditId = await AuditService.startAudit({
  projectId: "proj_123",
  startUrl: "https://example.com",
  config: {
    maxPages: 500,
    lighthouseStrategy: "sample", // Options: "none" | "sample" | "full"
  },
  billingCustomer: {
    userId: "user_456",
    organizationId: "org_789",
  },
});

// Poll for progress
const status = await AuditService.getStatus(auditId, "proj_123");
// Returns: { phase: "crawling" | "lighthouse" | "completed", pagesCrawled: number, ... }

// Retrieve results when complete
const results = await AuditService.getResults(auditId, "proj_123");

```

## Summary

- **OpenSEO** performs site audits using a **durable Cloudflare workflow** split into five discrete, retryable phases.
- The **Discovery** phase parses [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) and sitemaps to build seed URLs while respecting byte budget limits.
- The **Crawl** phase implements BFS with strict same-origin and robots.txt validation, using a custom user agent and 2 MiB HTML size limits.
- **Lighthouse** testing runs in batches of 10 URLs across mobile and desktop devices when enabled.
- **Multipage checks** detect cross-site SEO issues like duplicate titles and orphan pages before finalization.
- All data persists to **D1 or Postgres** via `AuditRepository`, with intermediate state stored in **KV** for real-time progress updates.

## Frequently Asked Questions

### How does OpenSEO respect robots.txt directives?

According to the source code in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts), the crawler validates every URL against the parsed [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) rules using `robots.isAllowed` before adding it to the crawl queue. The Discovery phase first fetches and parses [`robots.txt`](https://github.com/every-app/open-seo/blob/main/robots.txt) in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts), making those rules available to the BFS crawl logic that filters URLs through `isCrawlableUrl` checks.

### What happens when OpenSEO encounters bot protection or crawling errors?

The `crawlPage` function in [`src/server/workflows/site-audit-workflow-helpers.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/site-audit-workflow-helpers.ts) detects bot challenges through HTTP status codes (401, 403, 429, 503) and body content markers defined in `CHALLENGE_BODY_MARKERS`. When blocked, the crawler records the specific status code and classification without extracting HTML content, allowing the audit to continue with other URLs while preserving error metadata for the final report.

### How does the workflow handle partial failures during long audits?

OpenSEO wraps each logical phase in `pgStep` calls (visible in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)), which leverage Cloudflare Durable Objects to create durable execution steps. If a specific step fails—for example, a Lighthouse batch timing out—the workflow retries that specific step without re-executing previously completed crawl work. The error handling in [`SiteAuditWorkflow.ts`](https://github.com/every-app/open-seo/blob/main/SiteAuditWorkflow.ts) captures exceptions, reports them to PostHog, marks the audit as failed in the database, and re-throws to maintain accurate state tracking.

### What is the maximum crawl size OpenSEO supports?

The `maxPages` configuration parameter controls crawl limits, with the system designed to respect the 1 MiB step output limit (`SITEMAP_SEED_BYTE_BUDGET`) for sitemap seeds and 2 MiB HTML size limits per page. The BFS algorithm in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) continuously checks `summaries.length < maxPages` to enforce these boundaries, while batch processing via `runCrawlBatch` ensures memory remains bounded even for large-scale audits.