# How OpenSEO's Site Audit Workflow Handles Crawling and Lighthouse Analysis

> Discover how OpenSEO's site audit workflow efficiently handles crawling and Lighthouse analysis using a multi-phase Cloudflare Workers approach for optimal performance.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-14

---

**OpenSEO runs site audits as a multi-phase Cloudflare Workers workflow that chunks crawling to stay within platform limits, then samples crawled pages for DataForSEO-powered Lighthouse analysis.**

OpenSEO is an open-source SEO platform built by every-app that orchestrates site audits through durable Cloudflare Workers workflows. Its architecture splits the audit pipeline into distinct phases—discovery, crawling, and Lighthouse analysis—each designed to handle resource constraints while providing real-time feedback. This article walks through the crawling and Lighthouse implementation, citing exact source locations and providing runnable code examples.

## How the Crawling Phase Works

The crawl phase in OpenSEO processes URLs in bounded chunks to respect Cloudflare Workers' execution limits and memory constraints. Rather than crawling an entire site in one invocation, the workflow uses a durable **scratchpad** object to maintain state across chunk executions.

### Chunked Crawl Architecture

The crawling logic lives in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts). The `runCrawlPhase` function loops while URLs remain in the frontier and the page budget (`maxPages`) isn't exhausted:

```typescript
// Excerpt from runCrawlPhase showing the chunk loop
while (pending > 0 && attemptedTotal < params.maxPages) {
  const result = await pgStep(step, `crawl-chunk-${chunkNo}`, CRAWL_CHUNK_STEP,
    () => runCrawlChunk({ ...params, chunkNo, attemptedBefore: attemptedTotal }));
  attemptedTotal = result.attempted;
  pending = result.pending;
}

```

Each chunk handles up to **`CHUNK_TARGET_PAGES = 200`** URLs and has a soft timeout of **`CHUNK_SOFT_DEADLINE_MS = 90_000`** milliseconds. These constants bound both memory usage and execution time per durable step.

### Per-Chunk Execution Flow

The `runCrawlChunk` function in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) follows this pattern:

1. **Claim URLs** from the scratchpad via `scratchpad.claimChunk`
2. **Fetch pages** using `crawlPage` with dynamic window sizing
3. **Buffer results** and flush to the database in sub-batches of **`PERSIST_BATCH_SIZE = 25`**
4. **Apply back-pressure** when more than **`MAX_QUEUED_PERSIST_BATCHES = 2`** persist operations are pending

Results are persisted through `persistCrawledPages`, which:

- Assigns **deterministic row IDs** (`deterministicAuditRowId`) for idempotent retries
- Runs per-page issue reporters (`runPageReporters`)
- Writes pages and issues via `AuditRepository.insertCrawledBatch`
- Records discovered internal links back to the scratchpad (`scratchpad.recordBatch`)

### Frontier Expansion and Link Discovery

For every crawled page, OpenSEO extracts internal links and applies three filters before queueing:

- **`isSameOrigin`** — same protocol and host
- **`isCrawlableUrl`** — excludes non-HTTP schemes, fragments, etc.
- **robots.txt allowance** — respects crawl directives

The `shouldQueueCrawlLink` helper in [`src/server/workflows/siteAuditWorkflowCrawl.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowCrawl.ts) implements these checks. Discovered URLs are capped at **`MAX_STORED_LINKS_PER_PAGE = 500`** per page and **`MAX_DISCOVERED_PER_BATCH = 20_000`** per batch to prevent unbounded growth.

### Live Progress Reporting

Every persisted batch updates **`AuditProgressKV`** so the UI can display real-time stats including `pagesCrawled`, `pagesTotal`, and page title snippets. This implementation lives in [`src/server/lib/audit/progress-kv.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/progress-kv.ts).

## How the Lighthouse Phase Works

After crawling completes, `runLighthousePhase` in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) optionally runs Lighthouse analysis based on `config.lighthouseStrategy`.

### Sample Selection Strategy

The `selectLighthouseSample` function in [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) chooses up to **10 pages**:

- **Always includes** the start URL (homepage)
- **Plus one page per URL-template group** — for example, one from `/blog/*`, one from `/product/*`, etc.

Pages that didn't return a 2xx status are automatically excluded. The selection respects the configured strategy (`auto` vs `none`).

### Fetching Lighthouse Results

For each selected URL, the workflow creates a durable fetch batch (`lighthouse-fetch-N`). The `fetchLighthouseResult` function uses a DataForSEO client to request both **mobile and desktop** reports:

```typescript
// Simplified Lighthouse fetch flow
const fetched = await fetchLighthouseResult({
  url: page.url,
  strategy: "mobile", // and "desktop"
  dataforseoClient: createDataforseoClient(config),
});

```

Errors are caught and converted to result objects with `errorMessage`, so partial failures don't abort the entire phase.

### Storing Raw Payloads in R2

The `storeLighthouseResult` function uploads raw JSON to **Cloudflare R2** using this key format:

```

site-audit/<project>/<audit>/<pageId>-<strategy>.json

```

This preserves the complete Lighthouse payload for debugging while keeping the database lean. The function enriches results with the R2 key and payload size before returning.

### Persisting and Finalizing

`runLighthousePhase` batches stored results via `AuditRepository.insertLighthouseResults` and updates progress counters (`lighthouseCompleted`, `lighthouseFailed`). The total job count is `sample.length * 2` (mobile + desktop), displayed in the audit UI.

Both fetch and persist operations run inside **durable DB steps** (`LIGHTHOUSE_FETCH_STEP`, `LIGHTHOUSE_PERSIST_STEP`) for safe retries and failure visibility.

## Triggering Audits Programmatically

You can start a site audit with Lighthouse enabled through the internal MCP tool:

```typescript
import { startAudit } from "@/server/mcp/tools/site-audit-tools";

await startAudit({
  url: "https://example.com",
  maxPages: 50,
  runLighthouse: true,  // Enables Lighthouse with auto strategy
  // Additional options: maxDepth, respectRobotsTxt, etc.
});

```

To inspect Lighthouse sampling logic directly:

```typescript
import { selectLighthouseSample } from "@/server/lib/audit/lighthouse";

const sample = selectLighthouseSample(
  cachedPages,              // Pages from AuditRepository.getPagesForAudit
  "https://example.com",   // Start URL (homepage)
  "auto"                   // Strategy: "auto" | "none"
);

```

And to store results manually:

```typescript
import { storeLighthouseResult } from "@/server/lib/audit/lighthouse";

const stored = await storeLighthouseResult({
  projectId,
  auditId,
  fetched: lighthouseFetchResult,
});
console.log("Stored at R2 key:", stored.r2Key);

```

## Summary

- **Crawling is chunked** to respect Cloudflare Workers limits, with 200 URLs per chunk and 90-second soft deadlines
- **Progress is durable** via a scratchpad object that survives retries, with deterministic IDs for idempotent persistence
- **Link discovery is bounded** by same-origin checks, robots.txt compliance, and hard caps on stored links per page
- **Lighthouse samples intelligently**—homepage plus representative pages from each URL template group, up to 10 total
- **Raw payloads go to R2** while scores and metadata live in the database, balancing completeness with query performance
- **All phases use durable steps** for fault tolerance, with live KV progress updates for UI feedback

## Frequently Asked Questions

### What triggers a Lighthouse analysis in OpenSEO?

Lighthouse runs after crawling completes when `runLighthouse: true` is passed to `startAudit` (or equivalent API). The `runLighthousePhase` function in [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) checks `config.lighthouseStrategy !== "none"` before proceeding. Even when enabled, only 2xx pages are eligible for sampling.

### How does OpenSEO prevent crawling from exceeding Workers limits?

The workflow splits crawling into chunks of `CHUNK_TARGET_PAGES` (200) URLs, each executing as a separate durable step. A soft deadline of 90 seconds per chunk prevents timeouts, while `PERSIST_BATCH_SIZE` (25) and `MAX_QUEUED_PERSIST_BATCHES` (2) throttle database writes to control memory pressure.

### Where are Lighthouse results stored and for how long?

Raw JSON payloads upload to **Cloudflare R2** under the path `site-audit/<project>/<audit>/<pageId>-<strategy>.json` as implemented in `storeLighthouseResult`. The parsed scores and R2 key references persist in the application's database. R2 retention policy depends on your Cloudflare account configuration; the open-source repository doesn't specify automatic deletion.

### Can I customize which pages get Lighthouse analysis?

The current `auto` strategy in `selectLighthouseSample` automatically picks the homepage plus one page per URL template group. For custom selection, you'd need to modify [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) or pass pre-filtered pages to the sampling function. The repository doesn't expose user-configurable sampling rules through the public API.