DataForSEO API Error Handling and Retry Logic in Open-SEO: A Deep Dive

Open-SEO implements a resilient, multi-layered error handling pipeline for DataForSEO API calls that automatically retries transient failures, classifies errors by business impact, and preserves billing accuracy even when tasks fail after charging.

The every-app/open-seo repository isolates all DataForSEO interactions behind a thin, well-tested abstraction layer. This design ensures that transient network issues and upstream service disruptions are handled gracefully while maintaining precise metering for billed operations. Understanding these patterns is essential for developers integrating third-party SEO data APIs into production workflows.

Core Error Handling Architecture

The foundation of Open-SEO's DataForSEO integration resides in src/server/lib/dataforseo/core.ts. This module exports factory functions that wrap the native fetch API with authentication, timeouts, and retry logic.

Authenticated Fetch with Built-in Retry

The createAuthenticatedFetch function constructs a request wrapper that enforces a 60-second timeout and implements automatic retries for idempotent read operations. When the DataForSEO API returns a server error (5xx), the wrapper enters a retry loop governed by two constants:

• DATAFORSEO_MAX_RETRIES = 2 (total attempts: initial + 2 retries)
• DATAFORSEO_RETRY_BACKOFF_MS = 250 (linear back-off multiplier)

The back-off delay calculates as 250ms × (attempt + 1), producing intervals of 250ms, 500ms, and 750ms for successive retries. This bounded exponential strategy prevents thundering-herd problems while allowing brief upstream outages to resolve.

import { serpApi } from "@/server/lib/dataforseo/core";

// Client automatically retries on 5xx with configured back-off
const api = serpApi();
const response = await api.task_post({
  target: "example.com",
  keyword: "seo tools",
  language_code: "en",
  location_code: 2840,
});

Error Classification Strategy

Rather than throwing generic HTTP errors, the fetch factory accepts an optional DataforseoErrorClassifier callback. This function receives the HTTP status code, raw response body, and request path, returning a product-specific AppError or null to trigger default handling.

In src/server/lib/dataforseo/core.ts (lines 90-106), unclassified errors fall back to a rich generic payload containing the full response context. This pattern allows different API consumers to map the same DataForSEO error to domain-specific exceptions—for example, converting a 403 response into DATAFORSEO_AUTH_FAILED or a rate limit into RATE_LIMITED.

import type { DataforseoErrorClassifier } from "@/server/lib/dataforseo/core";
import { AppError } from "@/server/lib/errors";

const classifier: DataforseoErrorClassifier = (status, body, path) => {
  if (status === 403) {
    return new AppError("DATAFORSEO_AUTH_FAILED", "Invalid API credentials");
  }
  return null; // Defer to default error factory
};

const api = serpApi(classifier);

Task-Level Validation and Billing Protection

Successful HTTP responses from DataForSEO return a JSON envelope containing one or more tasks, each with its own status code. The src/server/lib/dataforseo/envelope.ts module provides assertOk and related utilities to validate these envelopes while safeguarding billing integrity.

The assertOk Validation Pipeline

The assertOk function implements a five-stage validation protocol:

1. Top-level status check: Verifies the envelope's status_code equals 20000; otherwise applies the error classifier or raises a generic exception.
2. Task existence: Confirms at least one task exists in the response array.
3. Task-level status: Validates the first task reports 20000 success.
4. Billing-aware failures: If a task failed but consumed credits (indicated by cost metadata), throws DataforseoChargedTaskError preserving the cost and path fields for metering.
5. Empty results handling: When treatNoResultsAsEmpty is true, converts "No Search Results" (40501) into an empty success rather than an error.

import { assertOk } from "@/server/lib/dataforseo/envelope";

try {
  const task = assertOk(response, {
    classify: myClassifier,
    treatNoResultsAsEmpty: true,
  });
  // Process task.result...
} catch (e) {
  if (e instanceof DataforseoChargedTaskError) {
    // e.billing contains { path, costUsd } for metering
    await recordCharge(e.billing);
    throw e; // Re-throw or handle based on business logic
  }
}

Billing Extraction and Charged Task Errors

The tryBuildTaskBilling helper (lines 70-84 in src/server/lib/dataforseo/envelope.ts) validates envelope fields (path, cost, result_count) using Zod schemas. When assertOk detects a non-20000 task status but finds valid cost metadata, it constructs a DataforseoChargedTaskError. This ensures that failed operations are still metered accurately before error propagation, preventing revenue leakage.

Invalid Field Diagnostics

For validation failures containing "Invalid Field: 'fieldName'" in the status_message, the describeInvalidField utility (lines 92-112) cross-references the original request payload (task.data). It appends the offending key-value pair to the error message, transforming opaque API errors into actionable debugging information.

Retry Strategies for External Services

Beyond DataForSEO-specific handling, Open-SEO demonstrates consistent retry patterns for auxiliary services.

Transient Upstream Retries (DataForSEO)

As implemented in src/server/lib/dataforseo/core.ts (lines 80-88), the retry loop strictly limits retries to 5xx status codes. The linear back-off calculation (DATAFORSEO_RETRY_BACKOFF_MS × (attempt + 1)) keeps latency predictable while the 60-second request timeout provides an absolute ceiling. This protects against brief downstream outages without exposing users to indefinite hanging requests.

Rate Limit Handling for Autumn Billing

The billing integration in src/serverFunctions/billing.ts handles HTTP 429 (Too Many Requests) from the Autumn service using a distinct strategy:

• Maximum attempts: AUTUMN_MAX_RETRIES = 3
• Respect Retry-After: Parses the header value, capping delay at AUTUMN_MAX_RETRY_DELAY_MS = 5000ms
• Progressive fallback: When the header is missing, applies AUTUMN_RETRY_BACKOFF_MS × (attempt + 1)

This pattern demonstrates how Open-SEO adapts retry logic to specific API contracts—honoring server-specified delays when available while maintaining defensive defaults.

// Simplified excerpt from fetchAutumnEventsPage
if (resp.status === 429 && attempt < AUTUMN_MAX_RETRIES) {
  const retryAfter = Number(resp.headers.get("Retry-After"));
  const delay = Number.isFinite(retryAfter)
    ? Math.min(retryAfter * 1000, AUTUMN_MAX_RETRY_DELAY_MS)
    : AUTUMN_RETRY_BACKOFF_MS * (attempt + 1);
  await new Promise(r => setTimeout(r, delay));
  continue;
}

Database Resilience Patterns

While not specific to DataForSEO, the repository's src/db/pg/retry.ts module illustrates the broader architectural commitment to resilience. The withQueryRetries utility wraps PostgreSQL queries, automatically retrying on transient connection errors (ECONNRESET) with jitter to prevent thundering-herd scenarios. This consistency across external HTTP APIs and internal database connections ensures system-wide stability.

Summary

• Automatic retries: DataForSEO API calls retry on 5xx errors up to 2 times with linear back-off (250ms increments) and a 60-second timeout.
• Structured error classification: The DataforseoErrorClassifier pattern allows custom mapping of HTTP status codes to domain-specific AppError types.
• Billing protection: assertOk validates task-level status codes and throws DataforseoChargedTaskError for failed tasks that consumed credits, preserving cost metadata.
• Diagnostic enrichment: Invalid field errors include the original request payload values for rapid debugging.
• Service-specific strategies: Rate-limited billing endpoints (Autumn) respect Retry-After headers with distinct constants (AUTUMN_MAX_RETRIES = 3).

Frequently Asked Questions

How does Open-SEO handle DataForSEO API rate limits?

Open-SEO primarily relies on DataForSEO's high rate limits and stable infrastructure; however, for the Autumn billing service (which uses distinct rate limiting), the implementation in src/serverFunctions/billing.ts specifically handles HTTP 429 responses. It parses the Retry-After header and delays subsequent requests up to a maximum of 5000ms, retrying up to 3 times before failing.

What happens when a DataForSEO task fails after billing?

When assertOk detects a task with a non-20000 status code that includes billing metadata (cost, path), it throws DataforseoChargedTaskError. This specialized error type preserves the billing information, allowing higher-level services to record the charge via meterDataforseoCall before deciding whether to propagate the error or attempt recovery.

Can I customize which DataForSEO errors are considered retryable?

Yes. While the built-in retry logic in createAuthenticatedFetch automatically retries only 5xx server errors, you can pass a custom DataforseoErrorClassifier to the client factory. This classifier receives the HTTP status, body, and path, allowing you to map specific error conditions (like 403 authentication failures) to appropriate AppError types before the retry logic evaluates them.

What is the maximum latency for a DataForSEO API call including retries?

The maximum latency is bounded by the 60-second request timeout configured in createAuthenticatedFetch. With DATAFORSEO_MAX_RETRIES = 2 and back-off delays of 250ms and 500ms, the worst-case scenario adds 750ms of retry overhead to the initial request duration, well within the timeout window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}