How Open-SEO Handles Caching for SEO-Related Data: R2 and KV Strategies

Open-SEO minimizes expensive third-party API calls by storing SEO data in Cloudflare R2 object storage and KV, using deterministic SHA-256 cache keys and configurable TTLs to balance freshness with performance.

The every-app/open-seo repository implements a dual-layer caching strategy to reduce latency and costs when fetching SEO metrics from providers like DataForSEO, Ahrefs, and Google Search Console. By leveraging Cloudflare's edge infrastructure, the system caches JSON responses in R2 for distributed persistence and KV for short-lived regional data. This architecture ensures that repeated requests for identical domain or keyword data bypass expensive external calls.

Core Caching Library in r2-cache.ts

The caching implementation centers on src/server/lib/r2-cache.ts, which provides three utilities for managing cache entries. At line 7, the file defines a CACHE_PREFIX constant (dataforseo-cache/) used to namespace all stored objects.

Deterministic Key Generation

The buildCacheKey function creates consistent identifiers by hashing the service name and parameters into a SHA-256 digest. This deterministic approach guarantees that identical requests always resolve to the same storage key.

const cacheKey = await buildCacheKey("serp:analysis", { domain, query, locale });

This pattern appears in src/server/features/keywords/services/research/serp.ts at line 2, where the SERP analysis service constructs unique keys for each keyword research query.

Cache Read and Write Operations

The getCached function (line 34) retrieves JSON values from R2, returning null when objects are missing or expired. The setCached function (line 51) writes payloads to R2 with TTL controls expressed in seconds, attaching the expiration via the Cache-Control header.

The Four-Step Caching Flow

Every SEO data request in Open-SEO follows a consistent retrieval pattern to minimize external API usage.

1. Generate the Cache Key

Services first construct a deterministic key that uniquely identifies the request parameters. For example, the SERP analysis feature builds keys incorporating domain, query, and locale values.

2. Attempt Cache Retrieval

Before contacting external APIs, the service checks for existing data. In src/server/features/keywords/services/research/serp.ts at lines 76-79, the implementation attempts to parse cached data using a Zod schema:

const cachedRaw = await getCached(cacheKey);
const cached = serpCacheSchema.safeParse(cachedRaw);
if (cached.success) return cached.data;

3. Fetch Fresh Data on Cache Miss

When getCached returns null or the data fails schema validation, the service proceeds to fetch fresh data from the appropriate third-party endpoint.

4. Write Results Back to Storage

After successful retrieval, the service stores the result for future requests. Line 94 of the SERP service demonstrates this fire-and-forget pattern:

void setCached(cacheKey, result, SERP_CACHE_TTL_SECONDS).catch(...);

Storage Backend Selection: R2 vs. KV

Open-SEO selects between Cloudflare R2 and KV based on data longevity and sharing requirements across regions.

Long-Term Persistence in Cloudflare R2

For SEO data requiring cross-region availability and longer retention—such as SERP results, domain overviews, and backlink profiles—the system uses R2 object storage. The CACHE_PREFIX ensures these objects are logically grouped under the dataforseo-cache/ namespace.

Regional Short-Term Storage in Cloudflare KV

For data that updates infrequently and doesn't require global synchronization, such as Ahrefs domain ratings, the system uses Cloudflare KV. This approach reduces latency for regional users while maintaining daily freshness.

In src/serverFunctions/ahrefs.ts at lines 78-86, the implementation checks KV before falling back to the API:

const cacheKey = `${CACHE_PREFIX}${domain}`;
const cached = await env.KV.get(cacheKey);
if (cached !== null) return parseCachedRating(cached);
await env.KV.put(cacheKey, JSON.stringify(dr), { expirationTtl: CACHE_TTL_SECONDS });

Implementation Patterns Across Services

SERP Analysis Caching

The keyword research service in src/server/features/keywords/services/research/serp.ts implements the full caching lifecycle. It defines SERP_CACHE_TTL_SECONDS at line 9 (set to 43,200 seconds, or 12 hours), ensuring SERP data remains fresh while reducing DataForSEO API calls.

Domain Overview Caching

src/server/features/domain/services/DomainService.ts follows an identical pattern. Lines 46-57 handle cache retrieval, while lines 93-95 store fresh data using DOMAIN_OVERVIEW_TTL_SECONDS, allowing domain metrics to be tuned independently from other SEO data.

Backlinks with Custom Abstraction

The backlinks feature introduces a BacklinksCache wrapper that encapsulates the core R2 utilities. In src/server/features/backlinks/services/backlinksServiceData.ts, lines 80-85 handle cache hits and lines 160-166 manage cache writes, providing type-safe methods for backlink-specific data structures.

TTL Strategy and Configuration

Open-SEO centralizes TTL values as constants to allow per-feature tuning of data freshness:

• SERP analysis: 12 hours (SERP_CACHE_TTL_SECONDS at line 9 of src/server/features/keywords/services/research/serp.ts)
• Ahrefs rating: 24 hours (CACHE_TTL_SECONDS at line 17 of src/serverFunctions/ahrefs.ts)
• Domain overview: Configured via DOMAIN_OVERVIEW_TTL_SECONDS at line 93 of src/server/features/domain/services/DomainService.ts
• General R2 data: Controlled by the CACHE_TTL object in src/server/lib/r2-cache.ts at line 7

Summary

Open-SEO's caching for SEO-related data delivers significant performance and cost benefits through these key mechanisms:

• Dual-backend architecture utilizing Cloudflare R2 for distributed persistence and KV for regional edge caching
• Deterministic key generation via SHA-256 hashing that ensures identical API requests share cache entries
• Schema-validated retrieval using Zod to ensure type safety when reading cached JSON
• Configurable TTL constants allowing each SEO feature to tune freshness windows independently
• Fire-and-forget writes that prevent cache updates from blocking response delivery

Frequently Asked Questions

What storage backends does Open-SEO use for caching SEO data?

Open-SEO uses Cloudflare R2 object storage for long-lived, globally distributed SEO data like SERP results and backlink profiles. For short-lived data that doesn't require cross-region synchronization, such as Ahrefs domain ratings, it uses Cloudflare KV with expiration TTLs.

How does Open-SEO generate cache keys to ensure consistency?

The system uses the buildCacheKey utility in src/server/lib/r2-cache.ts to create SHA-256 hashes from service names and request parameters. This deterministic approach ensures that identical queries (same domain, keyword, and locale) always produce the same cache key, maximizing cache hit rates.

What is the default cache duration for SERP analysis data?

SERP analysis data caches for 12 hours (43,200 seconds) as defined by SERP_CACHE_TTL_SECONDS in src/server/features/keywords/services/research/serp.ts. This duration balances the need for fresh search results with the goal of minimizing expensive DataForSEO API calls.

How does the caching layer reduce third-party API costs?

By checking Cloudflare R2 or KV before initiating external requests, Open-SEO serves repeated queries from edge storage rather than billable third-party endpoints. This pattern eliminates redundant calls for identical domain lookups, keyword analyses, and rating checks, directly reducing API usage charges from providers like DataForSEO and Ahrefs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}