How OpenSEO's Site Audit Workflow Handles Crawling and Lighthouse Analysis

OpenSEO runs site audits as a multi-phase Cloudflare Workers workflow that chunks crawling to stay within platform limits, then samples crawled pages for DataForSEO-powered Lighthouse analysis.

OpenSEO is an open-source SEO platform built by every-app that orchestrates site audits through durable Cloudflare Workers workflows. Its architecture splits the audit pipeline into distinct phases—discovery, crawling, and Lighthouse analysis—each designed to handle resource constraints while providing real-time feedback. This article walks through the crawling and Lighthouse implementation, citing exact source locations and providing runnable code examples.

How the Crawling Phase Works

The crawl phase in OpenSEO processes URLs in bounded chunks to respect Cloudflare Workers' execution limits and memory constraints. Rather than crawling an entire site in one invocation, the workflow uses a durable scratchpad object to maintain state across chunk executions.

Chunked Crawl Architecture

The crawling logic lives in src/server/workflows/siteAuditWorkflowCrawl.ts. The runCrawlPhase function loops while URLs remain in the frontier and the page budget (maxPages) isn't exhausted:

// Excerpt from runCrawlPhase showing the chunk loop
while (pending > 0 && attemptedTotal < params.maxPages) {
  const result = await pgStep(step, `crawl-chunk-${chunkNo}`, CRAWL_CHUNK_STEP,
    () => runCrawlChunk({ ...params, chunkNo, attemptedBefore: attemptedTotal }));
  attemptedTotal = result.attempted;
  pending = result.pending;
}

Each chunk handles up to CHUNK_TARGET_PAGES = 200 URLs and has a soft timeout of CHUNK_SOFT_DEADLINE_MS = 90_000 milliseconds. These constants bound both memory usage and execution time per durable step.

Per-Chunk Execution Flow

The runCrawlChunk function in src/server/workflows/siteAuditWorkflowCrawl.ts follows this pattern:

1. Claim URLs from the scratchpad via scratchpad.claimChunk
2. Fetch pages using crawlPage with dynamic window sizing
3. Buffer results and flush to the database in sub-batches of PERSIST_BATCH_SIZE = 25
4. Apply back-pressure when more than MAX_QUEUED_PERSIST_BATCHES = 2 persist operations are pending

Results are persisted through persistCrawledPages, which:

• Assigns deterministic row IDs (deterministicAuditRowId) for idempotent retries
• Runs per-page issue reporters (runPageReporters)
• Writes pages and issues via AuditRepository.insertCrawledBatch
• Records discovered internal links back to the scratchpad (scratchpad.recordBatch)

Frontier Expansion and Link Discovery

For every crawled page, OpenSEO extracts internal links and applies three filters before queueing:

• isSameOrigin — same protocol and host
• isCrawlableUrl — excludes non-HTTP schemes, fragments, etc.
• robots.txt allowance — respects crawl directives

The shouldQueueCrawlLink helper in src/server/workflows/siteAuditWorkflowCrawl.ts implements these checks. Discovered URLs are capped at MAX_STORED_LINKS_PER_PAGE = 500 per page and MAX_DISCOVERED_PER_BATCH = 20_000 per batch to prevent unbounded growth.

Live Progress Reporting

Every persisted batch updates AuditProgressKV so the UI can display real-time stats including pagesCrawled, pagesTotal, and page title snippets. This implementation lives in src/server/lib/audit/progress-kv.ts.

How the Lighthouse Phase Works

After crawling completes, runLighthousePhase in src/server/workflows/siteAuditWorkflowPhases.ts optionally runs Lighthouse analysis based on config.lighthouseStrategy.

Sample Selection Strategy

The selectLighthouseSample function in src/server/lib/audit/lighthouse.ts chooses up to 10 pages:

• Always includes the start URL (homepage)
• Plus one page per URL-template group — for example, one from /blog/*, one from /product/*, etc.

Pages that didn't return a 2xx status are automatically excluded. The selection respects the configured strategy (auto vs none).

Fetching Lighthouse Results

For each selected URL, the workflow creates a durable fetch batch (lighthouse-fetch-N). The fetchLighthouseResult function uses a DataForSEO client to request both mobile and desktop reports:

// Simplified Lighthouse fetch flow
const fetched = await fetchLighthouseResult({
  url: page.url,
  strategy: "mobile", // and "desktop"
  dataforseoClient: createDataforseoClient(config),
});

Errors are caught and converted to result objects with errorMessage, so partial failures don't abort the entire phase.

Storing Raw Payloads in R2

The storeLighthouseResult function uploads raw JSON to Cloudflare R2 using this key format:

site-audit/<project>/<audit>/<pageId>-<strategy>.json

This preserves the complete Lighthouse payload for debugging while keeping the database lean. The function enriches results with the R2 key and payload size before returning.

Persisting and Finalizing

runLighthousePhase batches stored results via AuditRepository.insertLighthouseResults and updates progress counters (lighthouseCompleted, lighthouseFailed). The total job count is sample.length * 2 (mobile + desktop), displayed in the audit UI.

Both fetch and persist operations run inside durable DB steps (LIGHTHOUSE_FETCH_STEP, LIGHTHOUSE_PERSIST_STEP) for safe retries and failure visibility.

Triggering Audits Programmatically

You can start a site audit with Lighthouse enabled through the internal MCP tool:

import { startAudit } from "@/server/mcp/tools/site-audit-tools";

await startAudit({
  url: "https://example.com",
  maxPages: 50,
  runLighthouse: true,  // Enables Lighthouse with auto strategy
  // Additional options: maxDepth, respectRobotsTxt, etc.
});

To inspect Lighthouse sampling logic directly:

import { selectLighthouseSample } from "@/server/lib/audit/lighthouse";

const sample = selectLighthouseSample(
  cachedPages,              // Pages from AuditRepository.getPagesForAudit
  "https://example.com",   // Start URL (homepage)
  "auto"                   // Strategy: "auto" | "none"
);

And to store results manually:

import { storeLighthouseResult } from "@/server/lib/audit/lighthouse";

const stored = await storeLighthouseResult({
  projectId,
  auditId,
  fetched: lighthouseFetchResult,
});
console.log("Stored at R2 key:", stored.r2Key);

Summary

• Crawling is chunked to respect Cloudflare Workers limits, with 200 URLs per chunk and 90-second soft deadlines
• Progress is durable via a scratchpad object that survives retries, with deterministic IDs for idempotent persistence
• Link discovery is bounded by same-origin checks, robots.txt compliance, and hard caps on stored links per page
• Lighthouse samples intelligently—homepage plus representative pages from each URL template group, up to 10 total
• Raw payloads go to R2 while scores and metadata live in the database, balancing completeness with query performance
• All phases use durable steps for fault tolerance, with live KV progress updates for UI feedback

Frequently Asked Questions

What triggers a Lighthouse analysis in OpenSEO?

Lighthouse runs after crawling completes when runLighthouse: true is passed to startAudit (or equivalent API). The runLighthousePhase function in src/server/workflows/siteAuditWorkflowPhases.ts checks config.lighthouseStrategy !== "none" before proceeding. Even when enabled, only 2xx pages are eligible for sampling.

How does OpenSEO prevent crawling from exceeding Workers limits?

The workflow splits crawling into chunks of CHUNK_TARGET_PAGES (200) URLs, each executing as a separate durable step. A soft deadline of 90 seconds per chunk prevents timeouts, while PERSIST_BATCH_SIZE (25) and MAX_QUEUED_PERSIST_BATCHES (2) throttle database writes to control memory pressure.

Where are Lighthouse results stored and for how long?

Raw JSON payloads upload to Cloudflare R2 under the path site-audit/<project>/<audit>/<pageId>-<strategy>.json as implemented in storeLighthouseResult. The parsed scores and R2 key references persist in the application's database. R2 retention policy depends on your Cloudflare account configuration; the open-source repository doesn't specify automatic deletion.

Can I customize which pages get Lighthouse analysis?

The current auto strategy in selectLighthouseSample automatically picks the homepage plus one page per URL template group. For custom selection, you'd need to modify src/server/lib/audit/lighthouse.ts or pass pre-filtered pages to the sampling function. The repository doesn't expose user-configurable sampling rules through the public API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}