How OpenSEO Implements DataForSEO API Response Caching for Brand Lookups

OpenSEO caches DataForSEO LLM Mentions API responses in Cloudflare R2 using deterministic SHA-256 keys with a 24-hour TTL, validating cached data with Zod schemas and only storing successful, data-rich results.

OpenSEO (every-app/open-seo) implements a robust DataForSEO API caching layer to minimize redundant external calls and reduce latency for repeated brand lookup queries. By persisting LLM Mentions results in Cloudflare R2 with deterministic key generation, the application ensures that identical requests return instantly while maintaining data freshness through automatic expiration.

Deterministic Cache Key Generation

The caching strategy begins with a deterministic cache key that uniquely identifies each brand lookup request based on its parameters.

SHA-256 Hashing of Request Parameters

In src/server/features/ai-search/services/brandLookup.ts, the getBrandLookup function constructs a cache key by passing normalized parameters to buildCacheKey from src/server/lib/r2-cache.ts:

const cacheKey = await buildCacheKey("ai-search:brand-lookup", {
  organizationId: billingCustomer.organizationId,
  projectId: input.projectId,
  targetType: detected.type,
  // lower‑cased for Data for SEO matching
  targetValue: detected.value.toLowerCase(),
  // competitors are sorted and pipe‑joined so the same set yields the same key
  competitors: competitorGroups
    .map(g => g.detected.value.toLowerCase())
    .toSorted()
    .join("|"),
  locationCode: input.locationCode,
  languageCode: input.languageCode,
});

The buildCacheKey helper serializes these parameters alphabetically, hashes the resulting JSON string with SHA-256, and prefixes it to create a key in the format ai-search:brand-lookup:<hash>. This guarantees that identical search criteria—regardless of parameter order—resolve to the same R2 object.

Competitor Normalization for Cache Consistency

To prevent duplicate cache entries from semantically identical requests, OpenSEO normalizes competitor arrays by converting values to lowercase, sorting them alphabetically, and joining them with pipe delimiters. This ensures that the same set of competitors in different orders generates identical cache keys, maximizing cache hit rates for OpenSEO brand lookup caching.

Cache Validation and Retrieval

Before invoking the DataForSEO API, OpenSEO attempts to retrieve validated cached data from Cloudflare R2.

Zod Schema Validation

The cache read operation in brandLookup.ts uses brandLookupResultSchema (defined in src/types/schemas/ai-search.ts) to validate stored JSON payloads:

const cached = brandLookupResultSchema.safeParse(await getCached(cacheKey));
if (cached.success) {
  return { ...cached.data, query: input.query, resolvedTarget: detected.value };
}

If validation passes, the function returns the cached result immediately, bypassing the external API call entirely. This type-safe approach prevents corrupted or outdated data from being served to users.

Cloudflare R2 Storage

The getCached utility in src/server/lib/r2-cache.ts retrieves objects stored under the dataforseo-cache/ prefix. Each cached entry includes customMetadata containing an expiresAt timestamp (ISO string) that enables client-side expiration checks before returning data.

Selective Write Strategy and TTL

OpenSEO employs a defensive caching strategy that only persists successful, data-rich responses.

Fire-and-Forget Writes with waitUntil

After calling DataForSEO endpoints (aggregatedMetrics, topPages, mentionsSearch) for each platform (chat_gpt, google) and shaping the results, the system writes to R2 only when all platform calls succeed and the result contains data (result.hasData):

if (allSucceeded && result.hasData) {
  waitUntil(
    setCached(cacheKey, result, BRAND_LOOKUP_TTL_SECONDS).catch(err => {
      console.error("ai-search.brand-lookup.cache-write failed:", err);
    })
  );
}

The waitUntil pattern (native to Cloudflare Workers) allows the HTTP response to return without blocking on the storage operation, improving perceived latency while ensuring durability.

24-Hour Expiration Policy

The BRAND_LOOKUP_TTL_SECONDS constant defines a 24-hour TTL for all brand lookup caches. The setCached function in r2-cache.ts sets both the R2 object metadata and the expiresAt field. When getCached retrieves an object, it compares the current time against expiresAt and treats expired entries as cache misses, forcing a fresh DataForSEO fetch and ensuring daily data refresh.

Complete Implementation Example

The following example demonstrates how getBrandLookup abstracts the caching complexity:

import { getBrandLookup } from "@/server/features/ai-search/services/brandLookup";

/* Example call – the first request hits DataforSEO,
   subsequent calls within 24 h hit the R2 cache. */
const brandResult = await getBrandLookup(
  {
    query: "Acme Corp",
    projectId: "proj_123",
    locationCode: "US",
    languageCode: "en",
    competitors: [],          // optional competitor list
  },
  { organizationId: "org_456" } // billing context
);

// brandResult now contains aggregated metrics, top pages, mentions, etc.
console.log(brandResult.topQueries);

Summary

• Deterministic keys: SHA-256 hashes of normalized parameters (lowercased values, sorted competitors) ensure consistent cache keys in src/server/lib/r2-cache.ts.
• Schema validation: All cached data is validated against brandLookupResultSchema before use, preventing type errors.
• Conditional persistence: Only successful API responses with actual data (hasData) are cached, avoiding pollution from partial failures.
• 24-hour freshness: The BRAND_LOOKUP_TTL_SECONDS constant enforces daily refresh via expiresAt metadata checks in getCached.
• Non-blocking writes: The waitUntil wrapper ensures cache writes do not delay HTTP responses.

Frequently Asked Questions

How long does OpenSEO cache DataForSEO brand lookup responses?

OpenSEO caches brand lookup responses for 24 hours using the BRAND_LOOKUP_TTL_SECONDS constant. After this period, the expiresAt metadata in Cloudflare R2 invalidates the entry, forcing a fresh DataForSEO API call on the next request.

What happens if cached DataForSEO data fails Zod validation?

If brandLookupResultSchema.safeParse() fails in src/server/features/ai-search/services/brandLookup.ts, the function treats the entry as a cache miss and proceeds to call the DataForSEO API. This prevents serving malformed or schema-outdated data to users.

Why does OpenSEO use waitUntil for cache writes?

OpenSEO uses waitUntil to perform fire-and-forget writes to Cloudflare R2. This allows the HTTP response to return immediately without waiting for the storage operation to complete, reducing latency while ensuring the cache is updated asynchronously in the background.

How does OpenSEO ensure identical competitor lists generate the same cache key?

The system normalizes competitor values by converting them to lowercase, sorting them alphabetically with toSorted(), and joining them with pipe characters (|). This normalization happens in getBrandLookup before the buildCacheKey call, ensuring semantic equivalence produces identical SHA-256 hashes regardless of original array order.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}