# How Keyword Research Works with DataForSEO Endpoints in Open SEO

> Learn how Open SEO leverages DataForSEO keyword endpoints for powerful keyword research. Discover secure authentication, normalized responses, and usage telemetry.

- Repository: [Every App/open-seo](https://github.com/every-app/open-seo)
- Tags: how-to-guide
- Published: 2026-08-07

---

**Open SEO proxies keyword research requests through a secure server layer that authenticates with DataForSEO using HTTP Basic Auth, normalizes complex API responses into a typed `Keyword` interface, and logs usage telemetry for billing enforcement.**

Open SEO is an open-source SEO platform that integrates with DataForSEO (DFSEO) to deliver comprehensive keyword analysis capabilities. The implementation wraps DFSEO's keyword endpoints in a type-safe architecture that handles authentication, data transformation, and error normalization. This article examines the complete request lifecycle from the React frontend through the server functions to the DFSEO API, referencing the actual source code in the `every-app/open-seo` repository.

## Architecture Overview: The Seven-Step Data Flow

The keyword research feature operates as a thin wrapper around the DFSEO keyword-analysis API. The pipeline flows through seven distinct stages:

1. **Client Request** – The React frontend calls the internal route `POST /api/keywords` using TanStack Query.
2. **Payload Construction** – The server function in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts) assembles a DFSEO request containing the target domain, language, location, seed keywords, and pagination parameters.
3. **Authentication** – The server creates an HTTP Basic Auth header using the `DATAFORSEO_API_KEY` environment variable.
4. **API Transmission** – The request is sent to `https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live` (or the related-keywords endpoint depending on the UI tab).
5. **Response Normalization** – DFSEO's complex JSON payload is flattened to match the internal `Keyword` type defined in [`src/types/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/types/keywords.ts).
6. **Error Mapping** – DFSEO errors (invalid keys, quota exceeded) are mapped to the project-wide error-code enum in [`src/shared/error-codes.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/error-codes.ts) and returned with HTTP 400/429 status codes.
7. **Telemetry Logging** – Successful requests are persisted to the `keyword_requests` table in [`src/db/schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts) for usage analytics and per-user request limiting via [`src/shared/billing.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/billing.ts).

## Client-Side Implementation with TanStack Query

The frontend consumes keyword data through a custom React hook that manages caching, background refetching, and error states. The hook calls the internal API route and expects an array of normalized `Keyword` objects in return.

```typescript
import { useQuery } from '@tanstack/react-query';

interface KeywordRequest {
  domain: string;
  language: string;
  location: string;
  seed: string;          // e.g. "blue shoes"
  page?: number;
  pageSize?: number;
}

/* Hook used in the Keyword‑Research page */
export function useKeywordResearch(req: KeywordRequest) {
  return useQuery(
    ['keyword-research', req],
    async () => {
      const resp = await fetch('/api/keywords', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify(req),
      });

      if (!resp.ok) {
        const err = await resp.json();
        throw new Error(err.message ?? 'Keyword request failed');
      }

      return (await resp.json()) as Keyword[];
    },
    { keepPreviousData: true, staleTime: 5 * 60_000 }
  );
}

```

## Server-Side Proxy and Authentication

The core integration logic resides in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts), which acts as a secure proxy between the client and DataForSEO's infrastructure.

### Constructing the DFSEO Request Payload

The server function extracts pagination parameters from the client request and calculates the appropriate offset for DFSEO's API. It supports both keyword suggestions and related-keyword endpoints based on the UI context.

### HTTP Basic Authentication Implementation

DataForSEO requires **HTTP Basic Auth** where the username is the API key and the password is empty. The server constructs the `Authorization` header server-side to ensure the `DATAFORSEO_API_KEY` environment variable never reaches the browser.

```typescript
import { json } from '@remix-run/node';
import { Keyword } from '../types/keywords';
import { DATAFORSEO_API_KEY } from '../../env'; // pulled from .env

export async function action({ request }) {
  const { domain, language, location, seed, page = 1, pageSize = 20 } = await request.json();

  const dfseoBody = {
    keywords: [seed],
    language,
    location_name: location,
    limit: pageSize,
    offset: (page - 1) * pageSize,
    // additional DFSEO‑specific flags…
  };

  const dfseoResp = await fetch(
    'https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live',
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        Authorization: `Basic ${btoa(`${DATAFORSEO_API_KEY}:`)}`,
      },
      body: JSON.stringify(dfseoBody),
    }
  );

  if (!dfseoResp.ok) {
    const err = await dfseoResp.json();
    return json({ message: err.message }, { status: dfseoResp.status });
  }

  const raw = await dfseoResp.json();
  const keywords: Keyword[] = raw.tasks.map((t: any) => ({
    keyword: t.keyword,
    volume: t.search_volume,
    cpc: t.cpc,
    difficulty: t.keyword_difficulty,
    competition: t.competition,
    serpFeatures: t.serp_features,
  }));

  // optional telemetry logging …
  return json(keywords);
}

```

## Response Normalization and Type Safety

DFSEO returns deeply nested JSON payloads containing extensive metadata. The server function extracts only the fields required by the UI and transforms them to match the internal domain model.

### The Keyword Type Definition

The normalized data structure is strictly typed in [`src/types/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/types/keywords.ts) to ensure consistency across the application:

```typescript
export type Keyword = {
  keyword: string;
  volume: number;             // monthly searches
  cpc: number;                // cost per click (USD)
  difficulty: number;         // 0‑100 scale
  competition: number;        // 0‑1 scale
  serpFeatures: string[];     // e.g. ["Featured Snippet","People Also Ask"]
};

```

### Field Mapping Strategy

The server maps DFSEO's `search_volume` to `volume`, `keyword_difficulty` to `difficulty`, and `serp_features` to `serpFeatures`. This abstraction allows the frontend to remain agnostic of third-party API naming conventions while providing TypeScript safety through the `Keyword` interface.

## Error Handling and Rate Limiting

The implementation includes robust error handling for DFSEO-specific failure modes and enforces usage limits through database tracking.

### Error Code Mapping

When DFSEO returns authentication failures, quota exceeded messages, or malformed request errors, the server catches these responses and maps them to standardized error codes defined in [`src/shared/error-codes.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/error-codes.ts). This allows the UI to display consistent, user-friendly error messages regardless of the underlying API provider.

### Usage Telemetry and Billing Enforcement

Every successful request is logged to the `keyword_requests` table defined in [`src/db/schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts). This telemetry enables the application to enforce per-user request limits through the billing logic in [`src/shared/billing.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/billing.ts), preventing abuse and supporting tiered subscription models.

## Summary

- **Open SEO** acts as a secure proxy to DataForSEO, with the core integration residing in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts).
- **Authentication** uses HTTP Basic Auth with the `DATAFORSEO_API_KEY` environment variable, constructed server-side to protect credentials.
- **Data normalization** transforms complex DFSEO responses into the clean `Keyword` type defined in [`src/types/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/types/keywords.ts), standardizing field names like `search_volume` to `volume`.
- **Error handling** maps DFSEO-specific errors to project-wide error codes from [`src/shared/error-codes.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/error-codes.ts) for consistent HTTP 400/429 responses.
- **Usage tracking** persists request metadata to the `keyword_requests` table in [`src/db/schema.ts`](https://github.com/every-app/open-seo/blob/main/src/db/schema.ts) to enforce billing limits via [`src/shared/billing.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/billing.ts).

## Frequently Asked Questions

### What specific DataForSEO endpoint does Open SEO use for keyword suggestions?

Open SEO calls `https://api.dataforseo.com/v3/keywords_data/keyword_suggestions/live` for generating keyword suggestions. Depending on the UI tab selected by the user, it may alternatively call the related-keywords endpoint. Both endpoints are accessed through the same server function in [`src/serverFunctions/keywords.ts`](https://github.com/every-app/open-seo/blob/main/src/serverFunctions/keywords.ts).

### How does Open SEO secure the DataForSEO API key?

The API key is stored in the `DATAFORSEO_API_KEY` environment variable and is only accessed server-side. The server function creates the Basic Auth header using `btoa(`${DATAFORSEO_API_KEY}:`)`, ensuring the credential never reaches the client browser or appears in frontend bundles.

### What happens when DataForSEO returns a rate limit or authentication error?

The server function catches DFSEO error responses and maps them to standardized error codes defined in [`src/shared/error-codes.ts`](https://github.com/every-app/open-seo/blob/main/src/shared/error-codes.ts). These are returned to the client with appropriate HTTP status codes (typically 400 for malformed requests or 429 for quota exceeded), allowing the React UI to display contextual error messages to the user.

### How does Open SEO handle pagination for large keyword datasets?

The client sends `page` and `pageSize` parameters to `POST /api/keywords`. The server calculates the `offset` as `(page - 1) * pageSize` and passes both `limit` (pageSize) and `offset` to DFSEO's API. This enables efficient pagination through large result sets while maintaining TanStack Query's caching and background refetching capabilities on the frontend.