DataForSEO Client Architecture for Error Handling and Retries in Open-SEO

The every-app/open-seo repository implements a four-layer DataForSEO client architecture that isolates network transport in core.ts, normalizes API responses through envelope.ts, and delegates retry logic to the Autumn engine (autumn.ts) to handle transient 5xx/429 errors with exponential back-off while strictly preventing retries of charged task errors via DataforseoChargedTaskError.

The every-app/open-seo codebase provides a production-grade TypeScript client for the DataForSEO API that prioritizes reliability and cost safety. This article examines the DataForSEO client architecture for error handling and retries, detailing how the system separates transient network failures from provider-specific charged errors to prevent double-billing while maximizing request success rates.

The Four-Layer Architecture

The client is organized into distinct layers, each with a single responsibility regarding error propagation and recovery.

1. Core HTTP Wrapper (core.ts)

The foundation resides in src/server/lib/dataforseo/core.ts, which manages raw HTTP transport to api.dataforseo.com. This module adds required authentication headers, performs the fetch operation, and parses JSON responses. It detects transient HTTP errors—specifically 5xx server errors and 429 rate-limit responses—and extracts Retry-After headers when present. When transient failures occur, it throws TransientDataforseoError signals that the upstream retry engine consumes.

2. Envelope Parsing and Error Classification (envelope.ts)

Located at src/server/lib/dataforseo/envelope.ts, this layer normalizes DataForSEO's envelope structure (task_post, task_get, task_status) into typed results. It distinguishes between transient failures (network issues, rate limits) and charged failures (invalid parameters that still consume API credits). When the envelope contains a charged failure status code such as 20002 (Invalid field), the parser throws DataforseoChargedTaskError. This error class explicitly signals that the operation should not be retried, protecting users from accidental double-billing.

3. The Autumn Retry Engine (autumn.ts)

The retry logic lives in src/server/billing/autumn.ts, a generic retry utility used across the codebase. The Autumn engine wraps request functions in a configurable loop that respects exponential back-off with jitter. It retries only idempotent operations—primarily GET requests and task polling—against a configurable allowlist of status codes (default: ["429", "500", "502", "503", "504"]). The engine honors Retry-After headers when available and defaults to exponential back-off otherwise, capping attempts at maxAttempts (default 3) to bound latency.

4. Service-Level APIs (serp.ts, labs.ts, etc.)

Concrete implementations such as src/server/lib/dataforseo/serp.ts compose the lower layers into domain-specific methods like fetchSerpOrganic(). These modules import the core wrapper, envelope parser, and Autumn helper, ensuring every DataForSEO request automatically inherits uniform error handling and retry semantics without duplicating logic across the codebase.

Error Handling Flow

The following sequence illustrates how a SERP request moves through the architecture when encountering failures:

1. Client Initialization – createDataforseoClient() (in src/server/lib/dataforseo/client.ts) instantiates a client with API credentials and base URL configuration.

2. Request Execution – Service modules invoke core.request(), which performs the HTTP fetch. On 5xx or 429 responses, it throws TransientDataforseoError.

3. Retry Evaluation – The Autumn engine (autumn.check()) intercepts transient errors and evaluates them against retryCodes. Valid transient errors trigger exponential back-off; charged errors like DataforseoChargedTaskError bubble through uncaught by the retry loop.

4. Envelope Validation – For successful HTTP responses (200 OK), envelope.ts validates the DataForSEO payload. Invalid parameters result in DataforseoChargedTaskError, which immediately surface to the caller without retry attempts.

5. Workflow Consumption – Higher-level workflows (e.g., src/server/workflows/RankCheckWorkflow.ts) catch these errors. Charged failures mark the step as failed with user-facing messages; surviving transient errors after exhaustion of maxAttempts are logged as critical upstream failures.

Key Implementation Details

Transient vs. Charged Error Semantics

The architecture strictly separates error types to balance reliability against cost:

• TransientDataforseoError: Represents temporary infrastructure issues (network timeouts, 5xx, rate limits). These are safe to retry and are handled by Autumn.
• DataforseoChargedTaskError: Represents semantic failures (invalid location codes, malformed parameters). DataForSEO bills these requests regardless of failure, so the client immediately surfaces them to prevent duplicate charges.

Retry Configuration Parameters

The Autumn engine accepts per-call overrides:

• retryCodes: Array of HTTP status codes eligible for retry (default includes 429, 500, 502, 503, 504).
• maxAttempts: Total execution attempts including the initial call (default 3).
• initialDelayMs: Base delay for exponential back-off calculation.

Practical Code Examples

Creating a DataForSEO Client

import { createDataforseoClient } from '@/server/lib/dataforseo';

const client = createDataforseoClient({
  apiKey: process.env.DATAFORSEO_API_KEY!,
  baseUrl: 'https://api.dataforseo.com',
});

Executing a SERP Request with Automatic Retries

import { fetchSerpOrganic } from '@/server/lib/dataforseo/serp';

async function getOrganicResults(url: string, location: string) {
  // Automatically retries transient 5xx/429 errors via Autumn
  const payload = await fetchSerpOrganic(client, {
    target: url,
    location_code: location,
  });
  return payload;
}

Handling Charged and Transient Errors

import { DataforseoChargedTaskError } from '@/server/lib/dataforseo/envelope';

try {
  const results = await getOrganicResults('https://example.com', 'us');
} catch (err) {
  if (err instanceof DataforseoChargedTaskError) {
    // Charged failure: do not retry, surface to user
    logger.warn('Invalid DataForSEO parameters: %s', err.message);
    throw new AppError('INVALID_LOCATION', err.message);
  }
  // Transient errors already retried by Autumn;
  // reaching here indicates exhaustion of maxAttempts
  throw err;
}

Custom Retry Configuration

import { autumn } from '@/server/billing/autumn';

await autumn.check(
  async () => fetchSerpOrganic(client, { target: url, location_code: 'us' }),
  {
    retryCodes: ['429', '500'],
    maxAttempts: 5,
    initialDelayMs: 200,
  }
);

Summary

• The client architecture separates concerns across four layers: Core (core.ts), Envelope (envelope.ts), Retry Engine (autumn.ts), and Service APIs (serp.ts).
• Transient errors (5xx, 429) trigger exponential back-off via the Autumn engine, respecting Retry-After headers and maxAttempts limits.
• Charged errors (DataforseoChargedTaskError) bypass retry logic entirely to prevent double-billing for invalid requests.
• All DataForSEO interactions inherit uniform error handling through the service-level abstractions, ensuring consistent behavior across SERP, Labs, Backlinks, and Google Ads endpoints.

Frequently Asked Questions

How does the Open-SEO client prevent double-billing on failed requests?

The architecture uses DataforseoChargedTaskError thrown by src/server/lib/dataforseo/envelope.ts to identify failures that consume API credits (e.g., invalid location codes). The Autumn retry engine in src/server/billing/autumn.ts only catches TransientDataforseoError, ensuring charged failures immediately surface to the caller without retry attempts.

What transient errors trigger automatic retries?

The client retries HTTP status codes 429, 500, 502, 503, and 504 by default. These are caught in src/server/lib/dataforseo/core.ts and converted to TransientDataforseoError, which the autumn.check() wrapper retries using exponential back-off with jitter, honoring any Retry-After headers provided by DataForSEO.

Can I adjust retry behavior for specific API calls?

Yes. While the service modules (serp.ts, etc.) use sensible defaults, you can invoke autumn.check() directly with a custom configuration object specifying retryCodes, maxAttempts, and initialDelayMs to override the global retry policy for individual requests.

Where is the retry logic implemented if I need to debug timeout issues?

The central retry implementation resides in src/server/billing/autumn.ts. This file manages the retry loop, back-off calculation, and error classification. For DataForSEO-specific error translation (mapping HTTP responses to charged vs. transient errors), inspect src/server/lib/dataforseo/envelope.ts and src/server/lib/dataforseo/core.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}