# How the Open‑SEO Lighthouse Service Runs and Stores Performance Audits

> Discover how Open-SEO runs Lighthouse performance audits. Learn about its multi-phase workflow, DataforSEO integration, Cloudflare R2 storage, and database persistence for distilled metrics.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-18

---

**Open‑SEO executes Lighthouse performance audits through a multi‑phase workflow that samples crawled pages, fetches scores via DataforSEO, stores raw JSON reports in Cloudflare R2, and persists distilled metrics to the database.**

The open‑source **Open‑SEO** repository orchestrates Lighthouse audits as a core component of its site‑audit pipeline. This article breaks down exactly how the lighthouse service runs and stores performance audits, tracing the execution flow from trigger to persistence with direct references to the source code implementation.

## Triggering the Lighthouse Audit Phase

The audit lifecycle begins in **[`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts)** at the `runLighthousePhase` function (lines 75–97).

The phase activates when an audit configuration specifies a Lighthouse strategy other than `"none"`. Valid strategies include `"auto"`, which triggers automatic sampling and execution. The workflow uses `pgStep` wrappers throughout, enabling **checkpoint‑replay safety** if the process is interrupted.

## Selecting Pages for Audit

Not every crawled page receives a Lighthouse audit. The system optimizes costs and runtime through intelligent sampling.

The `selectLighthousePages` helper retrieves crawled pages from the database, then delegates to **`selectLighthouseSample`** in [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) (lines 94–103).

The sampling algorithm follows this logic:

- Always include the start page
- Add up to one page per **URL template**
- Cap the total at **10 pages maximum**

This produces a representative subset without redundant template testing.

## Fetching Performance Data from DataforSEO

For each selected URL, the workflow calls **`fetchLighthouseResult`** in [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) (lines 25–44).

The fetch process:

1. Creates a DataforSEO client via `createDataforseoClient`
2. Invokes the DataforSEO "live" Lighthouse endpoint
3. Transforms the response into a lightweight `LighthouseResult` object
4. Retains the raw JSON payload for storage
5. Catches errors and populates `errorMessage` fields instead of failing

Both **mobile** and **desktop** strategies execute in parallel using `Promise.all`, cutting latency approximately in half:

```typescript
// Fetch mobile & desktop results in parallel
const [mobile, desktop] = await Promise.all([
  fetchLighthouseResult(url, pageId, "mobile", billingCustomer),
  fetchLighthouseResult(url, pageId, "desktop", billingCustomer),
]);

```

Batches respect `LEGACY_LIGHTHOUSE_URL_BATCH_SIZE` (10 URLs) to keep checkpoint sizes manageable and memory usage bounded.

## Storing Raw Reports in Cloudflare R2

After fetching, **`storeLighthouseResult`** (same file, lines 74–92) handles persistence of the raw JSON payload.

The function uploads to **Cloudflare R2** via `putTextToR2` from [`src/server/lib/r2.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2.ts). The storage key follows a deterministic pattern:

```

site-audit/<projectId>/<auditId>/<pageId>-<strategy>.json

```

Returned metadata includes:
- `key` — the R2 object path
- `sizeBytes` — payload size for storage accounting

This separation of **raw archival** (R2) from **queryable metrics** (database) balances cost and access patterns.

## Persisting Distilled Results to the Database

The final persistence layer writes structured performance data through **`AuditRepository.insertLighthouseResults`** ([`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts)).

Stored fields include:
- Performance scores (overall and category)
- Core Web Vitals metric values
- R2 reference metadata for raw report retrieval
- Error states and diagnostic information

The workflow then updates audit progress flags:
- `lighthouseCompleted` — successful completion
- `lighthouseFailed` — terminal failure state

These updates flow through `AuditRepository.updateAuditProgress`, maintaining accurate job status for the UI and retry logic.

## Complete Code Example

This snippet demonstrates the full audit lifecycle for a single page:

```typescript
// 1️⃣ Select a sample of pages for Lighthouse (auto strategy)
const sampleUrls = selectLighthouseSample(crawledPages, startUrl, "auto");

// 2️⃣ Process each URL (batched internally)
for (const url of sampleUrls) {
  // Fetch mobile & desktop results in parallel
  const [mobile, desktop] = await Promise.all([
    fetchLighthouseResult(url, pageId, "mobile", billingCustomer),
    fetchLighthouseResult(url, pageId, "desktop", billingCustomer),
  ]);

  // 3️⃣ Persist raw payload to Cloudflare R2
  const storedMobile = await storeLighthouseResult({
    projectId,
    auditId,
    fetched: mobile,
  });
  const storedDesktop = await storeLighthouseResult({
    projectId,
    auditId,
    fetched: desktop,
  });

  // 4️⃣ Save the distilled results to the database
  await AuditRepository.insertLighthouseResults(auditId, [
    storedMobile,
    storedDesktop,
  ]);
}

```

## Key Source Files and Responsibilities

| File | Role |
|------|------|
| [`src/server/workflows/siteAuditWorkflowPhases.ts`](https://github.com/every-app/open-seo/blob/main/src/server/workflows/siteAuditWorkflowPhases.ts) | Orchestrates the Lighthouse phase, batching, and progress updates |
| [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) | Contains `fetchLighthouseResult`, `storeLighthouseResult`, and `selectLighthouseSample` |
| [`src/server/lib/dataforseo/index.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/dataforseo/index.ts) | Provides `createDataforseoClient` for the DataforSEO live endpoint |
| [`src/server/lib/r2.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2.ts) | Implements `putTextToR2` for Cloudflare R2 uploads |
| [`src/server/features/audit/repositories/AuditRepository.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/audit/repositories/AuditRepository.ts) | Persists Lighthouse results and updates audit progress in PostgreSQL |

## Summary

- **Trigger**: The `runLighthousePhase` workflow activates when Lighthouse strategy is not `"none"`
- **Sampling**: `selectLighthouseSample` picks the start page plus template representatives, maximum 10 pages
- **Fetching**: `fetchLighthouseResult` calls DataforSEO in parallel for mobile and desktop
- **Storage**: Raw JSON archives to Cloudflare R2 with structured keys; distilled metrics to PostgreSQL
- **Reliability**: `pgStep` wrappers enable resumable execution; progress tracking supports UI status and retries

## Frequently Asked Questions

### How does Open‑SEO decide which pages to run Lighthouse on?

The system uses `selectLighthouseSample` in [`src/server/lib/audit/lighthouse.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/audit/lighthouse.ts) to pick a representative subset: the start page, up to one page per URL template, with a hard cap of 10 pages total. This minimizes audit cost while maintaining coverage diversity.

### Where are the raw Lighthouse reports stored long‑term?

Raw JSON payloads upload to **Cloudflare R2** via `storeLighthouseResult`, using the key pattern `site-audit/<projectId>/<auditId>/<pageId>-<strategy>.json`. The database stores only distilled scores, metrics, and R2 references.

### What happens if a Lighthouse fetch fails?

`fetchLighthouseResult` catches errors and returns a `LighthouseResult` with `errorMessage` populated. The workflow continues processing other URLs and marks the audit with `lighthouseFailed` status rather than crashing.

### Can the workflow resume if interrupted?

Yes. All major steps wrap in `pgStep` calls from the workflow engine, creating durable checkpoints. If the process restarts, it replays from the last committed checkpoint rather than starting over.