How the R2 Caching Layer Works for Expensive API Calls in Open‑SEO

TLDR: Open‑SEO stores expensive third‑party API results in Cloudflare R2 using deterministic SHA‑256 cache keys and soft‑TTL metadata, enabling edge‑cached responses without repeatedly hitting external services.

The Open‑SEO repository implements a robust caching layer with R2 storage to minimize latency and costs when calling expensive external APIs like DataForSEO. By persisting JSON responses in Cloudflare R2 with configurable time‑to‑live (TTL) settings, the system serves subsequent requests directly from the edge rather than re‑fetching from the source. This architecture leverages soft‑expiry metadata and deterministic key generation to balance data freshness with performance.

Cache Key Generation and TTL Configuration

Building Deterministic Cache Keys with buildCacheKey

Located in src/server/lib/r2-cache.ts, the buildCacheKey function normalizes request parameters by sorting them alphabetically before hashing them with SHA‑256. This ensures that semantically identical API calls generate identical cache keys regardless of parameter order, preventing duplicate storage. The resulting hash is prefixed with dataforseo-cache/ to maintain a consistent namespace within the R2 bucket.

import { buildCacheKey } from "@/server/lib/r2-cache";

const key = await buildCacheKey(
  "keywords",
  { domain: "example.com", country: "us", language: "en" },
);
// → "dataforseo-cache:keywords:ab34f9c7e8…"

Configuring Time‑To‑Live in CACHE_TTL

The same file exports a CACHE_TTL constant that maps data categories to hard‑coded durations in seconds. For example, keyword‑research results are cached for 86,400 seconds (24 hours), while other data types may have different retention periods. These values determine how long the system considers a cached entry fresh before triggering a background refresh.

Reading and Writing Cached Responses

Storing Data with setCached

When an external API returns a fresh payload, the setCached helper serializes the JSON and writes it to R2 using env.R2.put. The function explicitly sets the Content‑Type header to application/json and attaches custom metadata including an expiresAt timestamp that marks the soft‑expiry time. This metadata allows the cache to implement TTL logic without relying on R2’s native expiration features.

import { setCached, CACHE_TTL } from "@/server/lib/r2-cache";

// After fetching fresh data from the external API
await setCached(cacheKey, apiResponse, CACHE_TTL.researchResult);
// Stored with metadata: { expiresAt: <timestamp 24h from now> }

Retrieving Data with getCached

On subsequent requests, getCached queries the R2 bucket using env.R2.get. It first inspects the expiresAt metadata; if the timestamp has passed, the entry is treated as a miss and discarded. Valid entries are parsed from JSON and returned immediately, bypassing the external API entirely and reducing response latency.

import { getCached } from "@/server/lib/r2-cache";

const cached = await getCached(cacheKey);
if (cached) {
  // Serve cached payload (already parsed JSON)
  return cached;
}
// If null, proceed to call the expensive API

Low‑Level R2 Utilities for Raw Payloads

Beyond the high‑level cache helpers, src/server/lib/r2.ts provides getJsonFromR2 and putTextToR2 for direct bucket access without TTL management. The Lighthouse workflow uses getJsonFromR2 to retrieve stored audit results, while putTextToR2 handles raw text uploads for scenarios that don’t require the soft‑expiry logic of the main caching layer.

// Direct raw-JSON retrieval (used by the Lighthouse workflow)
import { getJsonFromR2 } from "@/server/lib/r2";

const json = await getJsonFromR2("lighthouse:12345");

// Writing raw text payloads
import { putTextToR2 } from "@/server/lib/r2";
await putTextToR2("reports:daily", textContent);

Complete Workflow Example

The typical execution flow in Open‑SEO follows a fail‑fast pattern: generate the cache key, attempt to read from R2, return cached data if valid, otherwise execute the expensive API call, then write the fresh result back to R2 with the appropriate TTL. This creates a self‑healing cache that degrades gracefully to the source API when data expires or misses.

import { buildCacheKey, getCached, setCached, CACHE_TTL } from "@/server/lib/r2-cache";

async function getKeywordsData(params) {
  // 1. Build deterministic key
  const key = await buildCacheKey("keywords", params);
  
  // 2. Check R2 cache
  const cached = await getCached(key);
  if (cached) return cached;
  
  // 3. Miss: call expensive external API
  const fresh = await dataforseoClient.getKeywords(params);
  
  // 4. Store for 24 hours and return
  await setCached(key, fresh, CACHE_TTL.researchResult);
  return fresh;
}

Summary

• Open‑SEO implements a caching layer with R2 storage to cache expensive API calls at the Cloudflare edge.
• buildCacheKey in src/server/lib/r2-cache.ts creates deterministic SHA‑256 keys prefixed with dataforseo-cache/.
• CACHE_TTL defines hard‑coded expiration periods (e.g., 24 hours for keyword research results).
• setCached writes JSON to R2 with application/json content‑type and soft‑expiry expiresAt metadata.
• getCached validates metadata timestamps before serving cached data, falling back to source APIs on miss or expiry.
• src/server/lib/r2.ts provides low‑level helpers (getJsonFromR2, putTextToR2) for direct bucket operations without TTL logic.

Frequently Asked Questions

How does Open‑SEO determine if a cached R2 entry is still valid?

Instead of using R2’s native expiration, the system stores an expiresAt timestamp in the object’s custom metadata. When getCached retrieves an object from src/server/lib/r2-cache.ts, it compares this timestamp against the current time. If the timestamp has passed, the entry is treated as expired and the system fetches fresh data from the external API.

What hash algorithm does the caching layer use for cache keys?

The buildCacheKey function uses SHA‑256 to hash the normalized request parameters. Parameters are sorted alphabetically before hashing to ensure consistency regardless of the order they were provided, producing deterministic keys that prevent duplicate cache entries for identical requests.

How long does Open‑SEO cache keyword research results?

According to the CACHE_TTL constant in src/server/lib/r2-cache.ts, keyword research results are cached for 86,400 seconds, which equals 24 hours. Other data types may have different TTL values defined in the same configuration object.

Can I use the R2 caching layer for non‑JSON data?

Yes. While the main cache helpers (getCached and setCached) handle JSON specifically, the low‑level utilities in src/server/lib/r2.ts support any content type. putTextToR2 can store raw text payloads, and getJsonFromR2 retrieves raw strings that you can parse or process as needed, as demonstrated in the Lighthouse workflow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}