How OpenSEO Caches Lighthouse JSON Payloads in Cloudflare R2

OpenSEO caches Lighthouse JSON payloads in Cloudflare R2 using deterministic keys built from project, audit, page, and strategy identifiers, enabling fast retrieval without re-running expensive DataForSEO API calls.

The OpenSEO project (available at every-app/open-seo) implements a durable caching layer for Lighthouse audit data to minimize latency and reduce API costs. When running SEO audits, the application stores the complete raw JSON returned by DataForSEO in object storage, then retrieves lightweight slices of that data on demand.

The Three-Stage Caching Workflow

The caching implementation follows a clear fetch-store-retrieve pattern across three main phases. Each phase is handled by specific modules in the src/server/lib/ directory.

Fetching the Lighthouse Report from DataForSEO

The process begins with fetchLighthouseResult in src/server/lib/audit/lighthouse.ts (lines 25-50). This function calls the DataForSEO API and returns two distinct artifacts:

• A distilled LighthouseResult object containing processed metrics
• The full, unmodified JSON string (payloadJson)

Capturing the raw JSON at this stage ensures that no diagnostic data is lost before storage.

Persisting JSON to Cloudflare R2

When payloadJson is present, storeLighthouseResult (lines 74-92 in the same file) handles the upload. The function constructs a deterministic R2 key by combining the project ID, audit ID, page ID, and strategy (mobile/desktop).

import { fetchLighthouseResult, storeLighthouseResult } from '@/server/lib/audit/lighthouse';

// Execute the live Lighthouse check via DataForSEO
const fetchResult = await fetchLighthouseResult(
  'https://example.com',
  'page-123',
  'mobile',
  billingCustomerContext,
);

// Persist the raw JSON to R2 when available
const storedResult = await storeLighthouseResult({
  projectId: 'proj-abc',
  auditId: 'audit-987',
  fetched: fetchResult,
});

// storedResult.r2Key contains the storage path
// storedResult.size contains the byte count

The actual upload is performed by putTextToR2, defined in src/server/lib/r2.ts (lines 12-26). This utility writes the string to Cloudflare R2 and returns the key along with the stored object size, which is then saved alongside the LighthouseResult record.

Retrieving Cached Payloads on Demand

API routes access cached data through getJsonFromR2 in src/server/lib/r2.ts (lines 3-10). The server retrieves the raw JSON string using the stored r2Key, then passes it through readStoredLighthousePayload from src/server/lib/lighthousePayload.ts.

This parsing function extracts only the essential metadata and issue-level data, preventing large payloads from being transmitted to the client unnecessarily.

import { getJsonFromR2 } from '@/server/lib/r2';
import { readStoredLighthousePayload } from '@/server/lib/lighthousePayload';

// Retrieve using the key stored in the audit record
const rawJson = await getJsonFromR2(storedResult.r2Key);

// Parse into a trimmed, issue-focused structure
const payload = readStoredLighthousePayload(rawJson);

The API surface in src/serverFunctions/lighthouse.ts (lines 28-44) orchestrates these calls, checking for the existence of a cached payload before falling back to fresh fetches.

Key Files in the Caching Layer

Understanding the module boundaries helps when extending or debugging the cache:

• src/server/lib/audit/lighthouse.ts – Contains fetchLighthouseResult for DataForSEO integration and storeLighthouseResult for R2 key generation and storage logic.
• src/server/lib/r2.ts – Provides low-level storage primitives: putTextToR2 for writes and getJsonFromR2 for reads.
• src/server/lib/lighthousePayload.ts – Houses readStoredLighthousePayload for JSON parsing and buildLighthouseExportFile for export generation.
• src/serverFunctions/lighthouse.ts – Server function that coordinates retrieval and returns structured data to the frontend.

Why Deterministic Keys Matter

The R2 key generation uses deterministic components (projectId, auditId, pageId, strategy) rather than random UUIDs. This design ensures that identical audit configurations always map to the same storage location, providing natural deduplication and preventing redundant storage costs for repeated audits of the same page.

Summary

• OpenSEO uses Cloudflare R2 as durable object storage for Lighthouse JSON payloads returned by DataForSEO.
• The storeLighthouseResult function in src/server/lib/audit/lighthouse.ts generates deterministic keys based on project, audit, page, and strategy identifiers.
• putTextToR2 and getJsonFromR2 in src/server/lib/r2.ts handle the low-level write and read operations.
• readStoredLighthousePayload trims the raw JSON to essential issue data before sending it to clients.
• This caching strategy eliminates redundant API calls to DataForSEO, significantly reducing both latency and operational costs.

Frequently Asked Questions

How does OpenSEO handle cache invalidation for Lighthouse reports?

OpenSEO relies on deterministic key generation rather than time-based expiration. When a new audit runs with the same project, audit, page, and strategy parameters, the new payload overwrites the existing object in R2. For explicit cache busts, the application generates new audit IDs, which automatically create fresh storage keys.

What storage service does OpenSEO use for JSON payloads?

According to the source code in src/server/lib/r2.ts, OpenSEO uses Cloudflare R2 for object storage. The putTextToR2 function uploads the raw Lighthouse JSON string to R2, while getJsonFromR2 retrieves it using the stored key reference saved in the audit record.

Can I access the full Lighthouse JSON or only processed subsets?

The system stores the complete raw JSON returned by DataForSEO in R2. However, when retrieving data through readStoredLighthousePayload in src/server/lib/lighthousePayload.ts, the function extracts only issue-level details and essential metadata. To access the full raw payload, you would call getJsonFromR2 directly with the stored key.

Where is the caching logic implemented in the codebase?

The primary caching logic resides in src/server/lib/audit/lighthouse.ts for the fetch-and-store workflow, src/server/lib/r2.ts for storage operations, and src/serverFunctions/lighthouse.ts for the API integration that retrieves cached data for client requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}