# How the Backlink Analysis Service Fetches and Caches Data in Open-SEO

> Discover how Open-SEO's backlink analysis service uses a cache-first approach with Cloudflare R2 and DataForSEO API to fetch and cache backlink data efficiently for 6 hours.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: internals
- Published: 2026-08-18

---

**The Open-SEO backlink analysis service implements a cache-first architecture that checks Cloudflare R2 storage before calling the DataForSEO API, caching fresh results for 6 hours to minimize external requests.**

Every backlink lookup in the **open-seo** repository flows through the `BacklinksService` located at [`src/server/features/backlinks/services/BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/BacklinksService.ts). This service orchestrates deterministic cache-key generation, schema-validated cache reads, and conditional DataForSEO API fetches with automatic write-back caching.

## Cache-Key Construction

Before any data retrieval begins, the service constructs a **deterministic cache key** that uniquely identifies each request. The `buildCacheKey` and `buildPageCacheKey` helpers combine multiple parameters into a single string:

- Organization ID
- Target URL (domain or page)
- Scope (`domain` or `page`)
- Pagination (`page`, `pageSize`)
- Sorting (`sortField`, `sortOrder`)
- Optional filters and spam-filter flags

This logic appears in [`BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/BacklinksService.ts) lines 48-55 and 65-70, ensuring identical requests hit the same cache entry regardless of when or where they originate.

## Reading from the R2-Backed Cache

The service implements a **cache-first read strategy**. For every incoming request, it immediately calls `getCached` to check the R2-backed cache store.

Cached entries undergo strict validation using Zod schemas:

- `backlinksOverviewCacheSchema` — validates overview responses
- `backlinksRowsPageResultSchema` — validates paginated backlink rows
- `backlinksReferringDomainsResultSchema` — validates referring domain data

If validation passes, the cached data returns immediately. You can see this pattern in `profileBacklinksOverview` at lines 80-87, where the service short-circuits to return cached results when available.

## Fetching Fresh Data from DataForSEO

On a **cache miss**, the service instantiates a DataForSEO client via `createDataforseoClient` (configured with billing credentials) and calls the appropriate endpoint:

| Endpoint | Purpose | Method |
|----------|---------|--------|
| `summary` | High-level backlink metrics | `profileBacklinksOverview` |
| `rows` | Individual backlink records | `profileBacklinksRowsPage` |
| `referringDomains` | Domain-level aggregation | `profileReferringDomainsPage` |
| `domainPages` | Page-level breakdown | `profileDomainPages` |

These calls appear in [`BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/BacklinksService.ts) lines 97-108 and `profileBacklinksRowsPage` lines 41-55.

## Response Transformation and Write-Back Caching

After receiving raw DataForSEO responses, the service **normalizes data into internal shapes** through dedicated mapping functions:

- `buildOverviewResult` — constructs the overview model
- `mapBacklinksRows` — transforms individual backlink records
- `mapReferringDomainsRows` — formats domain-level data

The transformed result is then **written to cache** with a 6-hour TTL using `setCached`. TTL constants are defined in [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts):

- `BACKLINKS_OVERVIEW_TTL_SECONDS` — 6 hours for overview data
- `BACKLINKS_TAB_TTL_SECONDS` — 6 hours for tabular results

This write-back logic appears in lines 73-79, 87-94, and 118-124 of [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts). Cache write failures are logged but never block the response.

## Practical Usage Examples

### Fetching a Backlink Overview

```typescript
import { BacklinksService } from "@/server/features/backlinks/services/BacklinksService";

async function getOverview(domain: string, billingCustomer) {
  const input = { target: domain };
  const { overview } = await BacklinksService.profileOverview(
    input,
    billingCustomer,
  );

  return overview;
}

```

### Paginated Backlink Rows with Caching

```typescript
import { BacklinksService } from "@/server/features/backlinks/services/BacklinksService";

async function getBacklinkRows(domain: string, page = 1, pageSize = 50, billingCustomer) {
  const input = {
    target: domain,
    scope: "domain",
    page,
    pageSize,
    sortField: "firstSeen",
    sortOrder: "desc",
    filters: {},
    mode: "one_per_domain",
  };

  const result = await BacklinksService.profileBacklinksPage(
    input,
    billingCustomer,
    { hideSpam: false },
  );

  return result;
}

```

### Using a Custom Cache for Testing

```typescript
import { createBacklinksService } from "@/server/features/backlinks/services/BacklinksService";

const inMemoryCache = {
  store: new Map<string, any>(),
  async get(key) { return this.store.get(key); },
  async set(key, data, ttl) {
    this.store.set(key, data);
    setTimeout(() => this.store.delete(key), ttl * 1000);
  },
};

const TestBacklinksService = createBacklinksService(inMemoryCache);

```

The `createBacklinksService` factory function enables dependency injection of custom cache implementations, making the service fully testable without R2 dependencies.

## Key Architecture Files

| File | Responsibility |
|------|--------------|
| [`src/server/features/backlinks/services/BacklinksService.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/BacklinksService.ts) | Public façade with cache-key generation and method delegation |
| [`src/server/features/backlinks/services/backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/backlinksServiceData.ts) | Core implementation: cache operations, API calls, data mapping |
| [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts) | R2 storage interface (`getCached`, `setCached`) |
| [`src/server/lib/dataforseo.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/dataforseo.ts) | HTTP client for DataForSEO API endpoints |
| [`src/server/features/backlinks/services/backlinksOverviewSchema.ts`](https://github.com/every-app/open-seo/blob/main/src/server/features/backlinks/services/backlinksOverviewSchema.ts) | Zod schemas for cache validation |

## Summary

- **Cache-first strategy** eliminates redundant DataForSEO calls for identical requests within the 6-hour TTL window
- **Deterministic cache keys** encode all request parameters to prevent collision between different queries
- **Schema validation** ensures cache integrity and graceful degradation on corrupted entries
- **Configurable cache backend** supports R2 production storage and custom implementations for testing
- **6-hour TTL** balances data freshness with API cost control

## Frequently Asked Questions

### How long does Open-SEO cache backlink data?

Cached backlink data persists for **6 hours** as defined by `BACKLINKS_OVERVIEW_TTL_SECONDS` and `BACKLINKS_TAB_TTL_SECONDS` in [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts). After TTL expiration, the next request triggers a fresh DataForSEO API call.

### What happens if the DataForSEO API is unavailable?

The source analysis does not reveal explicit fallback logic for API failures. Based on the implementation pattern in [`backlinksServiceData.ts`](https://github.com/every-app/open-seo/blob/main/backlinksServiceData.ts), API errors would propagate to the caller since the service awaits `createDataforseoClient` calls without catch blocks for network-level retries.

### Can I disable caching for real-time backlink data?

The `BacklinksService` does not expose a cache-bypass flag in its public methods. To force fresh data, you would need to instantiate the service with a custom cache that always returns `null` from `getCached`, or directly modify the cache key construction to include a timestamp nonce.

### What cache storage does Open-SEO use in production?

Production deployments use **Cloudflare R2** via the `getCached` and `setCached` utilities in [`src/server/lib/r2-cache.ts`](https://github.com/every-app/open-seo/blob/main/src/server/lib/r2-cache.ts). This provides durable, globally distributed object storage with S3-compatible semantics.