How the DataForSEO Client Handles API Request Batching and Error Recovery in Open SEO

The DataForSEO client in Open SEO uses a meter wrapper that automatically batches bulk API requests into compliant chunk sizes while centralizing error handling to distinguish between charged errors (which still consume credits) and validation failures (which do not).

The Open SEO repository integrates with the DataForSEO API through a thin abstraction layer designed to manage high-volume data operations efficiently. This implementation tackles two critical challenges inherent to third-party SEO data providers: optimizing expensive network calls through intelligent batching and ensuring accurate billing attribution even when requests fail. Understanding how the DataForSEO client handles API request batching and error recovery reveals a robust pattern for managing metered external APIs in production TypeScript applications.

Lazy Loading and the Meter Wrapper Architecture

The client architecture prioritizes cold-start performance by deferring the loading of the ~3 MB dataforseo-client SDK until the first actual API invocation. In src/server/lib/dataforseo/client.ts, the loadDataforseoSections() function initializes the heavy dependency only when needed, keeping serverless function startup times minimal.

The core abstraction is the meter() wrapper function, which transforms raw DataForSEO section methods into metered operations. When createDataforseoClient() is invoked with a customer context, it constructs an object where every leaf method passes through meterDataforseoCall. This wrapper receives three key parameters: the customer billing context, a picker function targeting the real SDK method, and an optional default credit feature identifier.

Execution flow diverges based on deployment mode. If the server runs in hosted mode (where usage credits must be tracked), meterDataforseoCall intercepts the request to apply billing logic; otherwise, it returns raw data directly without overhead.

Batching Strategies for High-Volume Endpoints

The client implements endpoint-specific batching logic to respect DataForSEO's payload limits while maximizing throughput. Large datasets are automatically sliced into compliant chunks, with each chunk executed as an independent API call.

Keyword Metrics Batching (700 Keywords Per Request)

For keyword volume and metrics retrieval, the client defines a KEYWORD_METRICS_BATCH_SIZE constant set to 700 in src/server/lib/dataforseo/keyword-metrics.ts. The public helper fetchKeywordMetricsForList slices input arrays exceeding this threshold into chunks, invoking the underlying endpoint for each segment sequentially.

This approach ensures that requests containing 1,500 keywords, for example, automatically split into two batches of 700 and one batch of 100. Each batch maintains the same location and language parameters, with results aggregated before returning to the caller.

Rank Check Task Batching (100 Tasks Per Request)

The rank monitoring functionality leverages DataForSEO's ability to queue multiple tasks in a single POST operation. According to the implementation in src/server/lib/dataforseo/client.ts, the rankCheckTaskPost method supports posting up to 100 queued rank-check tasks simultaneously.

Because DataForSEO bills the entire task_post request as a single unit regardless of task count, the meter wrapper charges exactly one credit feature for the complete batch. This design optimizes cost efficiency when scheduling large-scale rank tracking operations across multiple keywords or domains.

Error Recovery and Credit Tracking Mechanisms

The meterDataforseoCall function serves as the central error handling chokepoint, implementing sophisticated logic to differentiate between billable failures and client-side validation errors.

Distinguishing Charged vs. Validation Errors

When exceptions occur inside meterDataforseoCall, the wrapper inspects the error type defined in src/server/lib/dataforseo/envelope.ts. If the error is a DataforseoChargedTaskError, indicating that DataForSEO has already billed the request despite the failure, the error is logged and credit usage is still recorded via trackDataforseoCost.

Conversely, for validation-type errors where error.isInvalidField is true and error.billing.costUsd <= 0, the wrapper throws a clean AppError with code "VALIDATION_ERROR". This allows calling code to handle malformed inputs without consuming user credits, distinguishing between "user made a mistake" and "provider failed after charging."

Unified Credit Tracking with trackDataforseoCost

Regardless of success or the specific error type, trackDataforseoCost is invoked to persist usage data. This function records the spent credits, provider identifier (dataforseo), API path, and cache status. The implementation ensures that even failed batches that consumed resources upstream are correctly attributed to the customer, maintaining billing integrity across all scenarios.

After successful execution and credit bookkeeping, the wrapper returns result.data (the unwrapped payload) to the original caller, abstracting away the metering complexity.

Implementation Examples

// Fetching keyword metrics for 1500 keywords (auto-batched into chunks of 700)
import { createDataforseoClient } from "@/server/lib/dataforseo/client";
import { getOrganizationCustomer } from "@/server/billing/subscription";

async function getMetrics(keywords: string[]) {
  const customer = await getOrganizationCustomer(/* org context */);
  const client = createDataforseoClient(customer);
  
  // Automatically splits into 2 batches of 700 + 1 batch of 100
  const rows = await client.keywords.adsSearchVolume({
    keywords,
    locationCode: 2840,          // United States
    locationName: "New York",
    languageCode: "en",
    creditFeature: "rank_tracking",
  });
  
  return rows; // Normalized rows aggregated from all batches
}

// Posting a batch of rank-check tasks (up to 100 tasks per call)
import { createDataforseoClient } from "@/server/lib/dataforseo/client";

async function queueRankChecks(tasks: RankCheckTask[]) {
  const client = createDataforseoClient(/* billing customer */);
  
  // Single credit charged for the entire batch of up to 100 tasks
  await client.serp.rankCheckTaskPost({
    tasks,
    creditFeature: "rank_tracking",
  });
}

Summary

• Lazy initialization keeps cold-starts fast by loading the 3 MB DataForSEO SDK only on first use via loadDataforseoSections().
• Automatic batching splits keyword lists into 700-item chunks and rank-check tasks into 100-task batches to respect API limits.
• Meter wrapper (meterDataforseoCall) centralizes all API calls to enforce credit tracking and error classification.
• Error differentiation distinguishes between DataforseoChargedTaskError (billable failures) and validation errors (non-billable), ensuring users only pay for actual provider usage.
• Unified tracking via trackDataforseoCost records every transaction, including failed requests that consumed credits, maintaining accurate billing records.

Frequently Asked Questions

What is the maximum number of keywords that can be processed in a single API call?

The DataForSEO client in Open SEO automatically batches keyword metrics requests into chunks of 700 keywords per API call. If you submit a list of 1,500 keywords, the fetchKeywordMetricsForList function in src/server/lib/dataforseo/keyword-metrics.ts splits this into three separate requests (700, 700, and 100) and aggregates the results transparently.

How does the client handle errors where DataForSEO has already charged for the request?

When a DataforseoChargedTaskError occurs—indicating the provider billed the request before failing—the meterDataforseoCall wrapper catches the exception, logs the failure, and still invokes trackDataforseoCost to record the spent credits. The error is then re-thrown so the application can handle the failure while maintaining accurate billing records.

What happens to validation errors that don't consume API credits?

For validation errors where error.isInvalidField is true and no cost was incurred (error.billing.costUsd <= 0), the wrapper throws a clean AppError with the code "VALIDATION_ERROR". This prevents the credit tracking system from recording a charge, ensuring users do not pay for malformed requests or invalid parameters.

Why does the rank-check task endpoint support exactly 100 tasks per batch?

DataForSEO's SERP API accepts up to 100 queued tasks in a single task_post request and bills the entire operation as one unit. The Open SEO client respects this limit in src/server/lib/dataforseo/client.ts to optimize cost efficiency, charging a single credit feature for the complete batch rather than individual charges per task.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}