How the Open-SEO Backlink Analysis Module Fetches and Caches Domain Authority Metrics

The Open-SEO backlink analysis module fetches domain authority metrics from DataForSEO, transforms the response into an internal format, and caches the results in Cloudflare R2 for six hours to reduce API costs and latency.

The every-app/open-seo repository provides a robust backlink analysis feature that aggregates domain authority data by combining external API calls with intelligent caching. Understanding how this system fetches and caches domain authority metrics reveals a fault-tolerant pattern you can adapt for any high-cost third-party data integration. The implementation spans target normalization, deterministic cache-key generation, and read-through caching with a configurable TTL.

Architecture of the Domain Authority Fetching and Caching Pipeline

The module follows a deterministic seven-step pipeline implemented in the BacklinksService layer:

1. Normalize the target using normalizeBacklinksTarget to ensure consistent API formatting.
2. Build a deterministic cache key via buildCacheKey in src/server/lib/r2-cache.ts.
3. Attempt cache read using getCached to check for existing data.
4. Fetch from DataForSEO on cache miss, calling backlinks.summary and optionally backlinks.history.
5. Transform the payload into the internal BacklinksOverviewResult shape.
6. Write to R2 using cacheValue (which wraps setCached) with a 6-hour TTL.
7. Return to UI via the server function getBacklinksOverview.

All logic resides in a thin service layer that both server-function endpoints and onboarding AI tools consume, ensuring consistent authority data across the application.

Target Normalization and Cache Key Generation

Before any network request, the system ensures input consistency and creates a stable identifier for the cache entry.

Normalizing the Input

In src/server/lib/dataforseo.ts, the normalizeBacklinksTarget function (used within profileBacklinksOverview) converts user input into a standardized apiTarget and human-readable display value. This prevents cache fragmentation caused by formatting differences (e.g., https://example.com vs example.com).

Building Deterministic Cache Keys

The buildCacheKey function in src/server/lib/r2-cache.ts generates a SHA-256 hash from sorted input fields. As implemented in src/server/features/backlinks/services/BacklinksService.ts:

const cacheKey = await buildCacheKey("backlinks:overview", {
  ...buildTargetCacheInput(input, billingCustomer),
});

The helper buildTargetCacheInput assembles organizationId, the normalized target, and scope (domain or page), ensuring identical requests produce identical keys like backlinks:overview:{hash}.

Reading from Cloudflare R2

The getCached utility in src/server/lib/r2-cache.ts attempts to retrieve the JSON object stored under the generated key. If the entry exists and has not exceeded the TTL, the service returns the cached BacklinksOverviewResult immediately, bypassing the external API entirely.

Fetching Domain Authority Metrics from DataForSEO

On a cache miss, the module calls DataForSEO's API through the client wrapper in src/server/lib/dataforseo.ts. The profileBacklinksOverview function in src/server/features/backlinks/services/backlinksServiceData.ts executes two potential calls:

• backlinks.summary: Supplies current authority scores including backlinks_spam_score and target_spam_score.
• backlinks.history: Provides historical authority trends when the scope is set to "domain".

These fields correspond to DataForSEO's 0-100 authority scale used to evaluate domain trust.

Caching Domain Authority Results in Cloudflare R2

Data Transformation

Once fetched, buildOverviewResult (in backlinksServiceData.ts) maps raw DataForSEO fields to the internal schema:

summary: {
  backlinksSpamScore: args.summary.backlinks_spam_score ?? null,
  targetSpamScore: args.summary.info?.target_spam_score ?? null,
}

This abstraction allows the UI to consume stable field names regardless of external API changes.

Writing to R2 with TTL

The cacheValue helper writes the transformed result back to Cloudflare R2:

await cacheValue(
  cache,
  cacheKey,
  { overview },
  BACKLINKS_OVERVIEW_TTL_SECONDS,
);

BACKLINKS_OVERVIEW_TTL_SECONDS is set to 6 hours, balancing data freshness with API rate limits and cost efficiency. This persists the data via setCached in r2-cache.ts.

Implementation Examples

Calling the Server Function (UI Integration)

Front-end code consumes the cached data through the getBacklinksOverview server function defined in src/serverFunctions/backlinks.ts:

import { getBacklinksOverview } from "@/serverFunctions/backlinks";

async function fetchDomainAuthority(domain: string) {
  const result = await getBacklinksOverview({
    target: domain,
    // scope: "domain" // optional
  });
  console.log("Authority (spam score):", result.summary.backlinksSpamScore);
  console.log("Target spam score:", result.summary.targetSpamScore);
}

This single call handles the entire cache lookup and refresh cycle internally.

Debugging Cache Keys

To inspect how keys are generated for specific inputs:

import { buildCacheKey } from "@/server/lib/r2-cache";
import { normalizeBacklinksTarget } from "@/server/lib/dataforseo";

async function debugCacheKey(domain: string, orgId: string) {
  const normalized = normalizeBacklinksTarget(domain, { scope: "domain" });
  const key = await buildCacheKey("backlinks:overview", {
    organizationId: orgId,
    target: normalized.apiTarget,
    scope: normalized.scope,
  });
  console.log("Cache key:", key);
}

Inspecting Cached Values

Directly retrieve cached entries using getCached:

import { getCached } from "@/server/lib/r2-cache";

async function inspectCache(key: string) {
  const cached = await getCached(key);
  console.log("Cached overview:", cached);
}

Summary

• The backlink analysis module in every-app/open-seo uses a deterministic 7-step pipeline to fetch and cache domain authority metrics efficiently.
• Cache keys are SHA-256 hashes built from normalized inputs including organizationId, target, and scope, ensuring stable lookups across requests.
• Cloudflare R2 stores the transformed BacklinksOverviewResult with a 6-hour TTL to minimize DataForSEO API costs while maintaining acceptable data freshness.
• DataForSEO provides authority scores (backlinks_spam_score, target_spam_score) via the backlinks.summary endpoint, mapped to internal fields by buildOverviewResult.
• The service layer (BacklinksService.ts and backlinksServiceData.ts) abstracts caching logic, allowing server functions like getBacklinksOverview to return fresh or cached data transparently.

Frequently Asked Questions

What is the cache TTL for domain authority metrics in Open-SEO?

The system caches domain authority metrics for 6 hours using the constant BACKLINKS_OVERVIEW_TTL_SECONDS. This TTL is defined in the backlink service configuration and passed to setCached in src/server/lib/r2-cache.ts, balancing data freshness with API cost management and rate-limit compliance.

How does the module prevent duplicate cache entries for the same domain?

The buildCacheKey function in src/server/lib/r2-cache.ts prevents duplication by SHA-256 hashing a normalized input object. Before hashing, normalizeBacklinksTarget in src/server/lib/dataforseo.ts standardizes the target format, while buildTargetCacheInput ensures consistent ordering of fields like organizationId, target, and scope. Identical inputs always generate identical cache keys.

Which DataForSEO endpoints provide the domain authority data?

According to the source code in backlinksServiceData.ts, the module consumes two DataForSEO endpoints: backlinks.summary for current authority scores (including backlinks_spam_score and target_spam_score) and backlinks.history for historical trend data when analyzing domain-wide metrics.

Where is the caching logic implemented in the codebase?

The caching logic is split across three primary files: src/server/lib/r2-cache.ts contains the generic getCached, setCached, and buildCacheKey utilities; src/server/features/backlinks/services/BacklinksService.ts builds cache keys specific to backlinks; and src/server/features/backlinks/services/backlinksServiceData.ts handles the actual read-through caching pattern when fetching authority data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}