How the Open‑SEO Lighthouse Service Runs and Stores Performance Audits

Open‑SEO executes Lighthouse performance audits through a multi‑phase workflow that samples crawled pages, fetches scores via DataforSEO, stores raw JSON reports in Cloudflare R2, and persists distilled metrics to the database.

The open‑source Open‑SEO repository orchestrates Lighthouse audits as a core component of its site‑audit pipeline. This article breaks down exactly how the lighthouse service runs and stores performance audits, tracing the execution flow from trigger to persistence with direct references to the source code implementation.

Triggering the Lighthouse Audit Phase

The audit lifecycle begins in src/server/workflows/siteAuditWorkflowPhases.ts at the runLighthousePhase function (lines 75–97).

The phase activates when an audit configuration specifies a Lighthouse strategy other than "none". Valid strategies include "auto", which triggers automatic sampling and execution. The workflow uses pgStep wrappers throughout, enabling checkpoint‑replay safety if the process is interrupted.

Selecting Pages for Audit

Not every crawled page receives a Lighthouse audit. The system optimizes costs and runtime through intelligent sampling.

The selectLighthousePages helper retrieves crawled pages from the database, then delegates to selectLighthouseSample in src/server/lib/audit/lighthouse.ts (lines 94–103).

The sampling algorithm follows this logic:

• Always include the start page
• Add up to one page per URL template
• Cap the total at 10 pages maximum

This produces a representative subset without redundant template testing.

Fetching Performance Data from DataforSEO

For each selected URL, the workflow calls fetchLighthouseResult in src/server/lib/audit/lighthouse.ts (lines 25–44).

The fetch process:

1. Creates a DataforSEO client via createDataforseoClient
2. Invokes the DataforSEO "live" Lighthouse endpoint
3. Transforms the response into a lightweight LighthouseResult object
4. Retains the raw JSON payload for storage
5. Catches errors and populates errorMessage fields instead of failing

Both mobile and desktop strategies execute in parallel using Promise.all, cutting latency approximately in half:

// Fetch mobile & desktop results in parallel
const [mobile, desktop] = await Promise.all([
  fetchLighthouseResult(url, pageId, "mobile", billingCustomer),
  fetchLighthouseResult(url, pageId, "desktop", billingCustomer),
]);

Batches respect LEGACY_LIGHTHOUSE_URL_BATCH_SIZE (10 URLs) to keep checkpoint sizes manageable and memory usage bounded.

Storing Raw Reports in Cloudflare R2

After fetching, storeLighthouseResult (same file, lines 74–92) handles persistence of the raw JSON payload.

The function uploads to Cloudflare R2 via putTextToR2 from src/server/lib/r2.ts. The storage key follows a deterministic pattern:

site-audit/<projectId>/<auditId>/<pageId>-<strategy>.json

Returned metadata includes:

• key — the R2 object path
• sizeBytes — payload size for storage accounting

This separation of raw archival (R2) from queryable metrics (database) balances cost and access patterns.

Persisting Distilled Results to the Database

The final persistence layer writes structured performance data through AuditRepository.insertLighthouseResults (src/server/features/audit/repositories/AuditRepository.ts).

Stored fields include:

• Performance scores (overall and category)
• Core Web Vitals metric values
• R2 reference metadata for raw report retrieval
• Error states and diagnostic information

The workflow then updates audit progress flags:

• lighthouseCompleted — successful completion
• lighthouseFailed — terminal failure state

These updates flow through AuditRepository.updateAuditProgress, maintaining accurate job status for the UI and retry logic.

Complete Code Example

This snippet demonstrates the full audit lifecycle for a single page:

// 1️⃣ Select a sample of pages for Lighthouse (auto strategy)
const sampleUrls = selectLighthouseSample(crawledPages, startUrl, "auto");

// 2️⃣ Process each URL (batched internally)
for (const url of sampleUrls) {
  // Fetch mobile & desktop results in parallel
  const [mobile, desktop] = await Promise.all([
    fetchLighthouseResult(url, pageId, "mobile", billingCustomer),
    fetchLighthouseResult(url, pageId, "desktop", billingCustomer),
  ]);

  // 3️⃣ Persist raw payload to Cloudflare R2
  const storedMobile = await storeLighthouseResult({
    projectId,
    auditId,
    fetched: mobile,
  });
  const storedDesktop = await storeLighthouseResult({
    projectId,
    auditId,
    fetched: desktop,
  });

  // 4️⃣ Save the distilled results to the database
  await AuditRepository.insertLighthouseResults(auditId, [
    storedMobile,
    storedDesktop,
  ]);
}

Key Source Files and Responsibilities

File	Role
src/server/workflows/siteAuditWorkflowPhases.ts	Orchestrates the Lighthouse phase, batching, and progress updates
src/server/lib/audit/lighthouse.ts	Contains fetchLighthouseResult, storeLighthouseResult, and selectLighthouseSample
src/server/lib/dataforseo/index.ts	Provides createDataforseoClient for the DataforSEO live endpoint
src/server/lib/r2.ts	Implements putTextToR2 for Cloudflare R2 uploads
src/server/features/audit/repositories/AuditRepository.ts	Persists Lighthouse results and updates audit progress in PostgreSQL

Summary

• Trigger: The runLighthousePhase workflow activates when Lighthouse strategy is not "none"
• Sampling: selectLighthouseSample picks the start page plus template representatives, maximum 10 pages
• Fetching: fetchLighthouseResult calls DataforSEO in parallel for mobile and desktop
• Storage: Raw JSON archives to Cloudflare R2 with structured keys; distilled metrics to PostgreSQL
• Reliability: pgStep wrappers enable resumable execution; progress tracking supports UI status and retries

Frequently Asked Questions

How does Open‑SEO decide which pages to run Lighthouse on?

The system uses selectLighthouseSample in src/server/lib/audit/lighthouse.ts to pick a representative subset: the start page, up to one page per URL template, with a hard cap of 10 pages total. This minimizes audit cost while maintaining coverage diversity.

Where are the raw Lighthouse reports stored long‑term?

Raw JSON payloads upload to Cloudflare R2 via storeLighthouseResult, using the key pattern site-audit/<projectId>/<auditId>/<pageId>-<strategy>.json. The database stores only distilled scores, metrics, and R2 references.

What happens if a Lighthouse fetch fails?

fetchLighthouseResult catches errors and returns a LighthouseResult with errorMessage populated. The workflow continues processing other URLs and marks the audit with lighthouseFailed status rather than crashing.

Can the workflow resume if interrupted?

Yes. All major steps wrap in pgStep calls from the workflow engine, creating durable checkpoints. If the process restarts, it replays from the last committed checkpoint rather than starting over.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}