# How the Open-SEO Backlink Analysis Module Fetches and Caches Domain Authority Metrics

> Learn how the Open-SEO backlink analysis module fetches domain authority metrics from DataForSEO, caches results in Cloudflare R2, reducing API costs and latency.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-07-30

---

**The Open-SEO backlink analysis module fetches domain authority metrics from DataForSEO, transforms the response into an internal format, and caches the results in Cloudflare R2 for six hours to reduce API costs and latency.**

The `every-app/open-seo` repository provides a robust backlink analysis feature that aggregates domain authority data by combining external API calls with intelligent caching. Understanding how this system fetches and caches domain authority metrics reveals a fault-tolerant pattern you can adapt for any high-cost third-party data integration. The implementation spans target normalization, deterministic cache-key generation, and read-through caching with a configurable TTL.

## Architecture of the Domain Authority Fetching and Caching Pipeline

The module follows a deterministic seven-step pipeline implemented in the `BacklinksService` layer:

1. **Normalize the target** using `normalizeBacklinksTarget` to ensure consistent API formatting.
2. **Build a deterministic cache key** via `buildCacheKey` in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts).
3. **Attempt cache read** using `getCached` to check for existing data.
4. **Fetch from DataForSEO** on cache miss, calling `backlinks.summary` and optionally `backlinks.history`.
5. **Transform the payload** into the internal `BacklinksOverviewResult` shape.
6. **Write to R2** using `cacheValue` (which wraps `setCached`) with a 6-hour TTL.
7. **Return to UI** via the server function `getBacklinksOverview`.

All logic resides in a thin service layer that both server-function endpoints and onboarding AI tools consume, ensuring consistent authority data across the application.

## Target Normalization and Cache Key Generation

Before any network request, the system ensures input consistency and creates a stable identifier for the cache entry.

### Normalizing the Input

In [`src/server/lib/dataforseo.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/dataforseo.ts), the `normalizeBacklinksTarget` function (used within `profileBacklinksOverview`) converts user input into a standardized `apiTarget` and human-readable display value. This prevents cache fragmentation caused by formatting differences (e.g., `https://example.com` vs `example.com`).

### Building Deterministic Cache Keys

The `buildCacheKey` function in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) generates a SHA-256 hash from sorted input fields. As implemented in [`src/server/features/backlinks/services/BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/BacklinksService.ts):

```typescript
const cacheKey = await buildCacheKey("backlinks:overview", {
  ...buildTargetCacheInput(input, billingCustomer),
});

```

The helper `buildTargetCacheInput` assembles `organizationId`, the normalized `target`, and `scope` (domain or page), ensuring identical requests produce identical keys like `backlinks:overview:{hash}`.

## Reading from Cloudflare R2

The `getCached` utility in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) attempts to retrieve the JSON object stored under the generated key. If the entry exists and has not exceeded the TTL, the service returns the cached `BacklinksOverviewResult` immediately, bypassing the external API entirely.

## Fetching Domain Authority Metrics from DataForSEO

On a cache miss, the module calls DataForSEO's API through the client wrapper in [`src/server/lib/dataforseo.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/dataforseo.ts). The `profileBacklinksOverview` function in [`src/server/features/backlinks/services/backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/backlinksServiceData.ts) executes two potential calls:

- **`backlinks.summary`**: Supplies current authority scores including `backlinks_spam_score` and `target_spam_score`.
- **`backlinks.history`**: Provides historical authority trends when the scope is set to "domain".

These fields correspond to DataForSEO's **0-100 authority scale** used to evaluate domain trust.

## Caching Domain Authority Results in Cloudflare R2

### Data Transformation

Once fetched, `buildOverviewResult` (in [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts)) maps raw DataForSEO fields to the internal schema:

```typescript
summary: {
  backlinksSpamScore: args.summary.backlinks_spam_score ?? null,
  targetSpamScore: args.summary.info?.target_spam_score ?? null,
}

```

This abstraction allows the UI to consume stable field names regardless of external API changes.

### Writing to R2 with TTL

The `cacheValue` helper writes the transformed result back to Cloudflare R2:

```typescript
await cacheValue(
  cache,
  cacheKey,
  { overview },
  BACKLINKS_OVERVIEW_TTL_SECONDS,
);

```

`BACKLINKS_OVERVIEW_TTL_SECONDS` is set to **6 hours**, balancing data freshness with API rate limits and cost efficiency. This persists the data via `setCached` in [`r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/r2-cache.ts).

## Implementation Examples

### Calling the Server Function (UI Integration)

Front-end code consumes the cached data through the `getBacklinksOverview` server function defined in [`src/serverFunctions/backlinks.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/backlinks.ts):

```typescript
import { getBacklinksOverview } from "@/serverFunctions/backlinks";

async function fetchDomainAuthority(domain: string) {
  const result = await getBacklinksOverview({
    target: domain,
    // scope: "domain" // optional
  });
  console.log("Authority (spam score):", result.summary.backlinksSpamScore);
  console.log("Target spam score:", result.summary.targetSpamScore);
}

```

This single call handles the entire cache lookup and refresh cycle internally.

### Debugging Cache Keys

To inspect how keys are generated for specific inputs:

```typescript
import { buildCacheKey } from "@/server/lib/r2-cache";
import { normalizeBacklinksTarget } from "@/server/lib/dataforseo";

async function debugCacheKey(domain: string, orgId: string) {
  const normalized = normalizeBacklinksTarget(domain, { scope: "domain" });
  const key = await buildCacheKey("backlinks:overview", {
    organizationId: orgId,
    target: normalized.apiTarget,
    scope: normalized.scope,
  });
  console.log("Cache key:", key);
}

```

### Inspecting Cached Values

Directly retrieve cached entries using `getCached`:

```typescript
import { getCached } from "@/server/lib/r2-cache";

async function inspectCache(key: string) {
  const cached = await getCached(key);
  console.log("Cached overview:", cached);
}

```

## Summary

- **The backlink analysis module** in `every-app/open-seo` uses a deterministic 7-step pipeline to fetch and cache domain authority metrics efficiently.
- **Cache keys** are SHA-256 hashes built from normalized inputs including `organizationId`, `target`, and `scope`, ensuring stable lookups across requests.
- **Cloudflare R2** stores the transformed `BacklinksOverviewResult` with a 6-hour TTL to minimize DataForSEO API costs while maintaining acceptable data freshness.
- **DataForSEO** provides authority scores (`backlinks_spam_score`, `target_spam_score`) via the `backlinks.summary` endpoint, mapped to internal fields by `buildOverviewResult`.
- **The service layer** ([`BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/BacklinksService.ts) and [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts)) abstracts caching logic, allowing server functions like `getBacklinksOverview` to return fresh or cached data transparently.

## Frequently Asked Questions

### What is the cache TTL for domain authority metrics in Open-SEO?

The system caches domain authority metrics for **6 hours** using the constant `BACKLINKS_OVERVIEW_TTL_SECONDS`. This TTL is defined in the backlink service configuration and passed to `setCached` in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts), balancing data freshness with API cost management and rate-limit compliance.

### How does the module prevent duplicate cache entries for the same domain?

The `buildCacheKey` function in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) prevents duplication by SHA-256 hashing a normalized input object. Before hashing, `normalizeBacklinksTarget` in [`src/server/lib/dataforseo.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/dataforseo.ts) standardizes the target format, while `buildTargetCacheInput` ensures consistent ordering of fields like `organizationId`, `target`, and `scope`. Identical inputs always generate identical cache keys.

### Which DataForSEO endpoints provide the domain authority data?

According to the source code in [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts), the module consumes two DataForSEO endpoints: `backlinks.summary` for current authority scores (including `backlinks_spam_score` and `target_spam_score`) and `backlinks.history` for historical trend data when analyzing domain-wide metrics.

### Where is the caching logic implemented in the codebase?

The caching logic is split across three primary files: [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) contains the generic `getCached`, `setCached`, and `buildCacheKey` utilities; [`src/server/features/backlinks/services/BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/BacklinksService.ts) builds cache keys specific to backlinks; and [`src/server/features/backlinks/services/backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/backlinksServiceData.ts) handles the actual read-through caching pattern when fetching authority data.