How OpenSEO Manages DataForSEO API Requests and Retries

OpenSEO implements a billing-aware retry strategy that retries transient 5xx errors only on idempotent read operations while strictly avoiding retries on billed POST requests to prevent double-charging.

OpenSEO, an open-source SEO platform maintained in the every-app/open-seo repository, implements a sophisticated request handling layer for DataForSEO (DFSEO) API interactions that balances resilience against transient failures with protection against accidental double-billing. The architecture separates idempotent read operations from costly write operations through distinct client factories and configurable retry policies defined in src/server/lib/dataforseo/core.ts.

Core Retry Architecture in core.ts

The foundation of OpenSEO's DataForSEO API request management resides in src/server/lib/dataforseo/core.ts, which exports an authenticated fetch wrapper with built-in retry logic.

The createAuthenticatedFetch Wrapper

The createAuthenticatedFetch function generates a fetch instance that handles authentication and retries. It accepts an optional classify parameter for error classification and a maxServerErrorRetries parameter that defaults to DATAFORSEO_MAX_RETRIES (2).

// src/server/lib/dataforseo/core.ts
const DATAFORSEO_MAX_RETRIES = 2;
const DATAFORSEO_RETRY_BACKOFF_MS = 250;

function createAuthenticatedFetch(
  classify?: DataforseoErrorClassifier,
  maxServerErrorRetries = DATAFORSEO_MAX_RETRIES,
) {
  return async (url: RequestInfo, init?: RequestInit): Promise<Response> => {
    const apiKey = await getRequiredEnvValue("DATAFORSEO_API_KEY");
    const headers = new Headers(init?.headers);
    headers.set("Authorization", `Basic ${apiKey}`);

    const signal = init?.signal ?? AbortSignal.timeout(DATAFORSEO_REQUEST_TIMEOUT_MS);

    for (let attempt = 0; ; attempt++) {
      const response = await fetch(url, { ...init, headers, signal });
      if (response.ok) return response;

      if (response.status >= 500 && attempt < maxServerErrorRetries) {
        await new Promise(r => setTimeout(r,
          DATAFORSEO_RETRY_BACKOFF_MS * (attempt + 1)));
        continue;
      }
      // Non-retryable errors transformed into AppError...
    }
  };
}

This implementation guarantees that the overall request never exceeds DATAFORSEO_REQUEST_TIMEOUT_MS (60,000ms) across all retry attempts.

Retry Configuration and Backoff Strategy

Only idempotent reads returning HTTP 5xx status codes trigger retries. The system uses a linear backoff calculated as 250ms × (attempt + 1), meaning delays of 250ms, 500ms, and 750ms for subsequent attempts when the default maximum of 2 retries is utilized.

Billing-Safe API Client Factories

OpenSEO creates separate API clients for different DataForSEO sections, explicitly controlling retry behavior based on billing implications.

Read-Only vs. Billed Operation Separation

In src/server/lib/dataforseo/core.ts, factory functions instantiate clients with appropriate retry policies:

// src/server/lib/dataforseo/core.ts
export const labsApi = () => new DataforseoLabsApi(API_BASE, http());
export const serpApi = () => new SerpApi(API_BASE, http());
export const businessDataTaskApi = () =>
  new BusinessDataApi(API_BASE, http(undefined, 0)); // No retries
export const onPageApi = () =>
  new OnPageApi(API_BASE, http(undefined, 0));       // No retries

Read operations (labsApi, serpApi) use the default retry configuration, allowing up to 2 retries on 5xx errors. Billed operations (businessDataTaskApi, onPageApi) pass maxServerErrorRetries = 0 to http(), ensuring failed paid requests never retry automatically.

Workflow Implementation in Rank Check Paths

The practical application of these retry policies appears in src/server/workflows/rankCheckPaths.ts, which orchestrates both live SERP checks and queued task workflows.

Live SERP Requests (Non-Retryable)

Live rank checks use the rankCheck method, which hits billed endpoints. These calls use the serpApi client but disable retries at the factory level for POST operations:

// src/server/workflows/rankCheckPaths.ts – checkBatchLive()
const settled = await Promise.allSettled(
  tasks.map(task =>
    ctx.client.serp.rankCheck({
      keyword: task.keyword,
      locationCode: ctx.locationCode,
      languageCode: ctx.languageCode,
      device: task.device,
      targetDomain: ctx.domain,
      depth: ctx.serpDepth,
    }).then(r => ({ ...r, device: task.device })))
);

Because rankCheck represents a charged live request, the underlying fetch wrapper returns errors immediately without retrying, preventing double-billing during transient DataForSEO outages.

Queued Task Creation (Non-Retryable)

Task posting operations also avoid retries to prevent duplicate charges:

// src/server/workflows/rankCheckPaths.ts – runQueuedCheck()
await pgStep(step, `post-tasks-${postIndex}`, SINGLE_ATTEMPT_STEP_CONFIG, async () =>
  ctx.client.serp.rankCheckTaskPost({ ... })
);

The SINGLE_ATTEMPT_STEP_CONFIG ensures exactly one attempt, while rankCheckTaskPost uses a client configured with maxServerErrorRetries = 0.

Task Polling (Retryable)

Conversely, result polling represents idempotent, free operations that safely support retries:

// src/server/workflows/rankCheckPaths.ts
const COLLECT_STEP_CONFIG = {
  retries: { limit: 2, delay: "10 seconds" as const },
  timeout: "5 minutes" as const,
};

outcome = await pgStep(step, `collect-${round}`, COLLECT_STEP_CONFIG,
  () => collectQueuedRound(ctx, batch));

The collectQueuedRound function executes multiple parallel fetchRankCheckTaskResult calls (effectively task_get endpoints). If network glitches occur, the workflow configuration retries after a 10-second delay, up to 2 additional attempts, without billing implications.

Testing the Retry Contract

The file src/server/lib/dataforseo/core.test.ts validates that the retry logic respects billing boundaries:

// src/server/lib/dataforseo/core.test.ts
it("does not retry a Lighthouse HTTP 5xx response", async () => {
  // Mock fetch → 5xx → ensure only one request is sent
});

These unit tests guarantee that paid operations like Lighthouse audits execute exactly once, while read operations receive resilient retry handling.

Summary

• Authenticated fetch wrapper: createAuthenticatedFetch in src/server/lib/dataforseo/core.ts centralizes API key management and retry logic with a 60-second timeout.
• Billing-aware retries: Only idempotent 5xx responses retry (max 2 attempts, 250ms linear backoff); billed POSTs use maxServerErrorRetries = 0.
• Factory pattern: labsApi and serpApi allow retries for reads; businessDataTaskApi and onPageApi explicitly disable retries for paid operations.
• Workflow safety: Live rankCheck and rankCheckTaskPost execute once only; free task_get polling leverages workflow-level retries with 10-second delays.
• Test coverage: core.test.ts enforces the contract that charged operations never retry.

Frequently Asked Questions

Does OpenSEO automatically retry all failed DataForSEO API requests?

No. OpenSEO selectively retries only idempotent read operations that return HTTP 5xx status codes. Billed POST requests—such as live SERP checks via rankCheck or task creation via rankCheckTaskPost—never retry automatically to prevent double-charging. The system distinguishes these through the maxServerErrorRetries parameter, set to 0 for paid operations and 2 (default) for free reads.

How does OpenSEO prevent double-charging on DataForSEO API calls?

The architecture prevents double-charging through explicit client factories in src/server/lib/dataforseo/core.ts. Functions like businessDataTaskApi and onPageApi instantiate clients with http(undefined, 0), where the second argument explicitly sets maxServerErrorRetries to zero. This ensures that transient 5xx errors on paid endpoints fail immediately rather than retrying, while workflows like rankCheckPaths.ts use SINGLE_ATTEMPT_STEP_CONFIG for task posting steps.

What is the retry backoff strategy for DataForSEO requests in OpenSEO?

For retriable operations, OpenSEO implements a linear backoff strategy defined by DATAFORSEO_RETRY_BACKOFF_MS (250ms). The delay calculates as 250ms × (attempt + 1), resulting in progressive waits of 250ms, 500ms, and 750ms across the maximum 2 retry attempts. This applies only to idempotent reads such as DataForSEO Labs or Keywords API GET requests, not to billed operations.

Which DataForSEO operations support automatic retries in OpenSEO?

Only read-only, idempotent operations support automatic retries. This includes DataForSEO Labs API queries, Keywords API lookups, and task_get polling operations within workflows (which use workflow-level retry configs with 10-second delays). Live SERP requests (rankCheck), task creation (rankCheckTaskPost), and Lighthouse audits explicitly disable retries through the createAuthenticatedFetch configuration to maintain billing integrity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}