How the Backlink Analysis Service Fetches and Caches Data in Open-SEO

The Open-SEO backlink analysis service implements a cache-first architecture that checks Cloudflare R2 storage before calling the DataForSEO API, caching fresh results for 6 hours to minimize external requests.

Every backlink lookup in the open-seo repository flows through the BacklinksService located at src/server/features/backlinks/services/BacklinksService.ts. This service orchestrates deterministic cache-key generation, schema-validated cache reads, and conditional DataForSEO API fetches with automatic write-back caching.

Cache-Key Construction

Before any data retrieval begins, the service constructs a deterministic cache key that uniquely identifies each request. The buildCacheKey and buildPageCacheKey helpers combine multiple parameters into a single string:

• Organization ID
• Target URL (domain or page)
• Scope (domain or page)
• Pagination (page, pageSize)
• Sorting (sortField, sortOrder)
• Optional filters and spam-filter flags

This logic appears in BacklinksService.ts lines 48-55 and 65-70, ensuring identical requests hit the same cache entry regardless of when or where they originate.

Reading from the R2-Backed Cache

The service implements a cache-first read strategy. For every incoming request, it immediately calls getCached to check the R2-backed cache store.

Cached entries undergo strict validation using Zod schemas:

• backlinksOverviewCacheSchema — validates overview responses
• backlinksRowsPageResultSchema — validates paginated backlink rows
• backlinksReferringDomainsResultSchema — validates referring domain data

If validation passes, the cached data returns immediately. You can see this pattern in profileBacklinksOverview at lines 80-87, where the service short-circuits to return cached results when available.

Fetching Fresh Data from DataForSEO

On a cache miss, the service instantiates a DataForSEO client via createDataforseoClient (configured with billing credentials) and calls the appropriate endpoint:

Endpoint	Purpose	Method
summary	High-level backlink metrics	profileBacklinksOverview
rows	Individual backlink records	profileBacklinksRowsPage
referringDomains	Domain-level aggregation	profileReferringDomainsPage
domainPages	Page-level breakdown	profileDomainPages

These calls appear in BacklinksService.ts lines 97-108 and profileBacklinksRowsPage lines 41-55.

Response Transformation and Write-Back Caching

After receiving raw DataForSEO responses, the service normalizes data into internal shapes through dedicated mapping functions:

• buildOverviewResult — constructs the overview model
• mapBacklinksRows — transforms individual backlink records
• mapReferringDomainsRows — formats domain-level data

The transformed result is then written to cache with a 6-hour TTL using setCached. TTL constants are defined in backlinksServiceData.ts:

• BACKLINKS_OVERVIEW_TTL_SECONDS — 6 hours for overview data
• BACKLINKS_TAB_TTL_SECONDS — 6 hours for tabular results

This write-back logic appears in lines 73-79, 87-94, and 118-124 of backlinksServiceData.ts. Cache write failures are logged but never block the response.

Practical Usage Examples

Fetching a Backlink Overview

import { BacklinksService } from "@/server/features/backlinks/services/BacklinksService";

async function getOverview(domain: string, billingCustomer) {
  const input = { target: domain };
  const { overview } = await BacklinksService.profileOverview(
    input,
    billingCustomer,
  );

  return overview;
}

Paginated Backlink Rows with Caching

import { BacklinksService } from "@/server/features/backlinks/services/BacklinksService";

async function getBacklinkRows(domain: string, page = 1, pageSize = 50, billingCustomer) {
  const input = {
    target: domain,
    scope: "domain",
    page,
    pageSize,
    sortField: "firstSeen",
    sortOrder: "desc",
    filters: {},
    mode: "one_per_domain",
  };

  const result = await BacklinksService.profileBacklinksPage(
    input,
    billingCustomer,
    { hideSpam: false },
  );

  return result;
}

Using a Custom Cache for Testing

import { createBacklinksService } from "@/server/features/backlinks/services/BacklinksService";

const inMemoryCache = {
  store: new Map<string, any>(),
  async get(key) { return this.store.get(key); },
  async set(key, data, ttl) {
    this.store.set(key, data);
    setTimeout(() => this.store.delete(key), ttl * 1000);
  },
};

const TestBacklinksService = createBacklinksService(inMemoryCache);

The createBacklinksService factory function enables dependency injection of custom cache implementations, making the service fully testable without R2 dependencies.

Key Architecture Files

File	Responsibility
src/server/features/backlinks/services/BacklinksService.ts	Public façade with cache-key generation and method delegation
src/server/features/backlinks/services/backlinksServiceData.ts	Core implementation: cache operations, API calls, data mapping
src/server/lib/r2-cache.ts	R2 storage interface (getCached, setCached)
src/server/lib/dataforseo.ts	HTTP client for DataForSEO API endpoints
src/server/features/backlinks/services/backlinksOverviewSchema.ts	Zod schemas for cache validation

Summary

• Cache-first strategy eliminates redundant DataForSEO calls for identical requests within the 6-hour TTL window
• Deterministic cache keys encode all request parameters to prevent collision between different queries
• Schema validation ensures cache integrity and graceful degradation on corrupted entries
• Configurable cache backend supports R2 production storage and custom implementations for testing
• 6-hour TTL balances data freshness with API cost control

Frequently Asked Questions

How long does Open-SEO cache backlink data?

Cached backlink data persists for 6 hours as defined by BACKLINKS_OVERVIEW_TTL_SECONDS and BACKLINKS_TAB_TTL_SECONDS in backlinksServiceData.ts. After TTL expiration, the next request triggers a fresh DataForSEO API call.

What happens if the DataForSEO API is unavailable?

The source analysis does not reveal explicit fallback logic for API failures. Based on the implementation pattern in backlinksServiceData.ts, API errors would propagate to the caller since the service awaits createDataforseoClient calls without catch blocks for network-level retries.

Can I disable caching for real-time backlink data?

The BacklinksService does not expose a cache-bypass flag in its public methods. To force fresh data, you would need to instantiate the service with a custom cache that always returns null from getCached, or directly modify the cache key construction to include a timestamp nonce.

What cache storage does Open-SEO use in production?

Production deployments use Cloudflare R2 via the getCached and setCached utilities in src/server/lib/r2-cache.ts. This provides durable, globally distributed object storage with S3-compatible semantics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}