How Keyword Research Works with DataForSEO Endpoints in Open SEO

Open SEO proxies keyword research requests through a secure server layer that authenticates with DataForSEO using HTTP Basic Auth, normalizes complex API responses into a typed Keyword interface, and logs usage telemetry for billing enforcement.

Open SEO is an open-source SEO platform that integrates with DataForSEO (DFSEO) to deliver comprehensive keyword analysis capabilities. The implementation wraps DFSEO's keyword endpoints in a type-safe architecture that handles authentication, data transformation, and error normalization. This article examines the complete request lifecycle from the React frontend through the server functions to the DFSEO API, referencing the actual source code in the every-app/open-seo repository.

Architecture Overview: The Seven-Step Data Flow

The keyword research feature operates as a thin wrapper around the DFSEO keyword-analysis API. The pipeline flows through seven distinct stages:

1. Client Request – The React frontend calls the internal route POST /api/keywords using TanStack Query.
2. Payload Construction – The server function in src/serverFunctions/keywords.ts assembles a DFSEO request containing the target domain, language, location, seed keywords, and pagination parameters.
3. Authentication – The server creates an HTTP Basic Auth header using the DATAFORSEO_API_KEY environment variable.
4. API Transmission – The request is sent to https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live (or the related-keywords endpoint depending on the UI tab).
5. Response Normalization – DFSEO's complex JSON payload is flattened to match the internal Keyword type defined in src/types/keywords.ts.
6. Error Mapping – DFSEO errors (invalid keys, quota exceeded) are mapped to the project-wide error-code enum in src/shared/error-codes.ts and returned with HTTP 400/429 status codes.
7. Telemetry Logging – Successful requests are persisted to the keyword_requests table in src/db/schema.ts for usage analytics and per-user request limiting via src/shared/billing.ts.

Client-Side Implementation with TanStack Query

The frontend consumes keyword data through a custom React hook that manages caching, background refetching, and error states. The hook calls the internal API route and expects an array of normalized Keyword objects in return.

import { useQuery } from '@tanstack/react-query';

interface KeywordRequest {
  domain: string;
  language: string;
  location: string;
  seed: string;          // e.g. "blue shoes"
  page?: number;
  pageSize?: number;
}

/* Hook used in the Keyword‑Research page */
export function useKeywordResearch(req: KeywordRequest) {
  return useQuery(
    ['keyword-research', req],
    async () => {
      const resp = await fetch('/api/keywords', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(req),
      });

      if (!resp.ok) {
        const err = await resp.json();
        throw new Error(err.message ?? 'Keyword request failed');
      }

      return (await resp.json()) as Keyword[];
    },
    { keepPreviousData: true, staleTime: 5 * 60_000 }
  );
}

Server-Side Proxy and Authentication

The core integration logic resides in src/serverFunctions/keywords.ts, which acts as a secure proxy between the client and DataForSEO's infrastructure.

Constructing the DFSEO Request Payload

The server function extracts pagination parameters from the client request and calculates the appropriate offset for DFSEO's API. It supports both keyword suggestions and related-keyword endpoints based on the UI context.

HTTP Basic Authentication Implementation

DataForSEO requires HTTP Basic Auth where the username is the API key and the password is empty. The server constructs the Authorization header server-side to ensure the DATAFORSEO_API_KEY environment variable never reaches the browser.

import { json } from '@remix-run/node';
import { Keyword } from '../types/keywords';
import { DATAFORSEO_API_KEY } from '../../env'; // pulled from .env

export async function action({ request }) {
  const { domain, language, location, seed, page = 1, pageSize = 20 } = await request.json();

  const dfseoBody = {
    keywords: [seed],
    language,
    location_name: location,
    limit: pageSize,
    offset: (page - 1) * pageSize,
    // additional DFSEO‑specific flags…
  };

  const dfseoResp = await fetch(
    'https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live',
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Basic ${btoa(`${DATAFORSEO_API_KEY}:`)}`,
      },
      body: JSON.stringify(dfseoBody),
    }
  );

  if (!dfseoResp.ok) {
    const err = await dfseoResp.json();
    return json({ message: err.message }, { status: dfseoResp.status });
  }

  const raw = await dfseoResp.json();
  const keywords: Keyword[] = raw.tasks.map((t: any) => ({
    keyword: t.keyword,
    volume: t.search_volume,
    cpc: t.cpc,
    difficulty: t.keyword_difficulty,
    competition: t.competition,
    serpFeatures: t.serp_features,
  }));

  // optional telemetry logging …
  return json(keywords);
}

Response Normalization and Type Safety

DFSEO returns deeply nested JSON payloads containing extensive metadata. The server function extracts only the fields required by the UI and transforms them to match the internal domain model.

The Keyword Type Definition

The normalized data structure is strictly typed in src/types/keywords.ts to ensure consistency across the application:

export type Keyword = {
  keyword: string;
  volume: number;             // monthly searches
  cpc: number;                // cost per click (USD)
  difficulty: number;         // 0‑100 scale
  competition: number;        // 0‑1 scale
  serpFeatures: string[];     // e.g. ["Featured Snippet","People Also Ask"]
};

Field Mapping Strategy

The server maps DFSEO's search_volume to volume, keyword_difficulty to difficulty, and serp_features to serpFeatures. This abstraction allows the frontend to remain agnostic of third-party API naming conventions while providing TypeScript safety through the Keyword interface.

Error Handling and Rate Limiting

The implementation includes robust error handling for DFSEO-specific failure modes and enforces usage limits through database tracking.

Error Code Mapping

When DFSEO returns authentication failures, quota exceeded messages, or malformed request errors, the server catches these responses and maps them to standardized error codes defined in src/shared/error-codes.ts. This allows the UI to display consistent, user-friendly error messages regardless of the underlying API provider.

Usage Telemetry and Billing Enforcement

Every successful request is logged to the keyword_requests table defined in src/db/schema.ts. This telemetry enables the application to enforce per-user request limits through the billing logic in src/shared/billing.ts, preventing abuse and supporting tiered subscription models.

Summary

• Open SEO acts as a secure proxy to DataForSEO, with the core integration residing in src/serverFunctions/keywords.ts.
• Authentication uses HTTP Basic Auth with the DATAFORSEO_API_KEY environment variable, constructed server-side to protect credentials.
• Data normalization transforms complex DFSEO responses into the clean Keyword type defined in src/types/keywords.ts, standardizing field names like search_volume to volume.
• Error handling maps DFSEO-specific errors to project-wide error codes from src/shared/error-codes.ts for consistent HTTP 400/429 responses.
• Usage tracking persists request metadata to the keyword_requests table in src/db/schema.ts to enforce billing limits via src/shared/billing.ts.

Frequently Asked Questions

What specific DataForSEO endpoint does Open SEO use for keyword suggestions?

Open SEO calls https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live for generating keyword suggestions. Depending on the UI tab selected by the user, it may alternatively call the related-keywords endpoint. Both endpoints are accessed through the same server function in src/serverFunctions/keywords.ts.

How does Open SEO secure the DataForSEO API key?

The API key is stored in the DATAFORSEO_API_KEY environment variable and is only accessed server-side. The server function creates the Basic Auth header using btoa(${DATAFORSEO_API_KEY}:), ensuring the credential never reaches the client browser or appears in frontend bundles.

What happens when DataForSEO returns a rate limit or authentication error?

The server function catches DFSEO error responses and maps them to standardized error codes defined in src/shared/error-codes.ts. These are returned to the client with appropriate HTTP status codes (typically 400 for malformed requests or 429 for quota exceeded), allowing the React UI to display contextual error messages to the user.

How does Open SEO handle pagination for large keyword datasets?

The client sends page and pageSize parameters to POST /api/keywords. The server calculates the offset as (page - 1) * pageSize and passes both limit (pageSize) and offset to DFSEO's API. This enables efficient pagination through large result sets while maintaining TanStack Query's caching and background refetching capabilities on the frontend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}